Trying to use StorageLoader with Stream Enrich without AWS

Binned Unsupported setup
6

@sandesh,

Here’s the “architecture” you should be using (any of the two should do):

  • Enrichment done in Kinesis Enrich:
... -> Stream enrich -> Kinesis S3 -> S3 -> EmrEtlRunner (shredding) -> PostgreSQL
  • Enrichment done in EMR
... -> Stream raw -> Kinesis S3 -> S3 -> EmrEtlRunner (enrich + shredding) -> PostgreSQL

A sample of the config.yml for R88 is here. The database target JSON configuration file is here.

Note the “targets” section was removed from YAML configuration in R88 and replaced with JSON configuration file.

You can refer to Lambda architecture to clarify this setup.