Error using StorageLoader to load data into Redshift

For engineers Storage targets
2

Hi @amitkhanal,

The correct configuration is

StreamCollector > Raw Stream >  Kinesis LZO S3 Sink > EmrEtlRunner > StorageLoader > Redshift

Note “Raw Stream” instead of “StreamEnrich”. The enrichment is done by EmrEtlRunner instead.

The error message “Cannot find atomic-events directory in shredded/good” can be explained by the fact StreamEnrich does not perform shredding. This functionality is a part of the “batch” pipeline.

Please, refer to the post How to setup a Lambda architecture for Snowplow for more details.

The other point is the R87 version of the runners is still in development/testing. I would stick to R83 instead for now.

–Ihor

2 Likes