PostgresDB: No data getting stored

For engineers Troubleshooting
1

I’ve installed Snowplow Opensource in AWS by following the quick start guide: Quick Start Installation Guide on AWS - Snowplow Docs
Installation completed without any errors.
Upon sending events using trackers, data is not getting stored in postgres db, neither enriched nor bad, though objects are present in s3 bucket for those events. events table is present in db but empty.
postgres-loader-server is also running for both enriched and bad.
Do I need to setup or install some additional things for data to get inserted into postgres?

2

Take a look at the cloudwatch logs for the postgres loader, this is often where I’ve seen the issue lie.

If you don’t see anything there (exceptions or errors stating the loader can’t load), then take a look at the other components and see if anything is erroring.

Usually when you’re in this state, it’s due to a slight configuration error (such as missing http:// on a url or something).

1 Like
3

Thanks for your reply, it was due to adding https:// instead of http:// to iglu dns.
After reinstalling it is working now but only good data i.e. atomic schema is getting populated and not atomic bad for bad events. Any idea what might be causing this issue, bad events are there in buckets. Didn’t find anything in logs. TIA

4

Hi @zameer

Problem with loading bad data sounds similar to issue we had in the past.

Could you please check values for fields like: collector, v_collector or loaderName in badrow JSONs stored in the S3?
If these look like snowplow-stream-collector-kinesis... instead of ssc-... then it may mean we violate schema definition (as table definitions in postgres are created based on JSON schemas defined in Iglu registry). Is there any error in loader logs indicated that?

The issue was fixed in 0.3.0 of terraform collector modules. It looks like quickstart-examples repository still uses 0.2.1 for AWS collector module.

We will plan necessary upgrades in quickstart repository and let you know as soon as the newest version is released. Meanwhile, you can even try to change version of snowplow-devops/collector-kinesis-ec2/aws in terraform to 0.3.0 locally and see if it helps!