Postgres Loader for Snowplow , not reading data from S3 enriched records

1

Postgres Loader for Snowplow , not reading data from S3 enriched records.

I use a Kinesis ScalaCollector -> EnrichEmrEtlrunner -> Amazon Kinesis Firehose -> S3 Enriched records -> PostgresqlLoader -> PostgresqlDB

The last step is not working. I see enriched records in S3.

2

@ckrishnamoorthy, is the other post of yours related to this - EtlEmrRunner fails at step [enrich] spark: Enriched HDFS -> S3 with error "Input path does not exist: hdfs://ip-172-31-10-133.us-east-2.compute.internal:8020/tmp/1d59dbe9-98f2-473f-8d65-288f5019fdca/files"?

3

Hi @ckrishnamoorthy,

Sorry for the confusion - the Postgres Loader is:

  1. Not released with final version yet (not has been announced - the docs page is there by mistake)
  2. Is not meant to load data from S3, but works directly with Kinesis/PubSub topics

What you’re referring to is probably RDB Loader, which is different application support Postgres (but very limited in its features)

4

Thanks @anton for clarifying. Actually i was looking for Postgreql loader, but mistook the source as S3. I do have a Kinesis stream that can be a source.
So when is this going to be available. If it is not avaiable, it should be removed from the documentation.

Thanks,
Krishna