Enriched event stream into Redshift using StorageLoader: Contract violation error

For engineers Storage targets
3

Glad you got it working @chipchip! Just a couple of notes for anybody else reading this:

Note 1. Going on your config.yml, I believe your setup is slightly different to what you described - rather than:

StreamCollector > StreamEnrich > kinesisS3SinkLZO > StorageLoader > Redshift

It looks like you are doing:

StreamCollector > raw stream > StreamEnrich
                             > kinesisS3SinkLZO > StorageLoader > Redshift

which is great - exactly per our Snowplow Lambda architecture.

Note 2. I don’t know why you would set the enriched archive to the same location as the raw in:

     raw: 
        in:
          - s3://kinesis-sink # e.g. s3://my-in-bucket
...
      enriched:
...
        archive: s3://kinesis-sink # Where to archive enriched events to, e.g. s3://my-archive-bucket/enriched

That is going to just store enriched events right where your raw payloads are landing, causing a ton of confusion.