We are pleased to announce Version 0.2.0 of the Snowplow Postgres Loader
The postgres loader is a young project in the Snowplow ecosystem, with potential to fulfil an important role in demo, QA, and low to mid volume pipelines. This 0.2.0 release still should not be considered for pipelines with requirements of scalability for high event volumes, or zero data loss. Nonetheless, this release takes us another step closer to the maturity of this project for production pipelines.
The blog post from the 0.1.0 release has more details about why you should be both excited and cautious about the postgres loader and its abilities.
Improvements over version 0.1.0
We added some desirable configuration options when streaming from kinesis: the ability to reduce costs by disabling enhanced fan out, and disabling CloudWatch metrics. On top of that we bumped a few library dependencies, and fixed a known bug affecting the yauaa enrichment.
Upgrading from 0.1.0
Config format changed
We switched the configuration file format from self-describing json to hocon, which is less strict but more adaptable and consistent how we configure other Snowplow pipeline applications. Some of the config field names have changed since 0.1.0, and this is part of our plan to be more consistent in naming conventions across our applications. The snowplow docs site has a detailed config reference guide, which will help you migrate to the new format.
Some field types changed from VARHAR to JSONB
We made a breaking change, but it only effects schemas with “exotic” field types, such as mixed enums of integers and strings, or object fields with no sub-fields specified. You can read more about this breaking change in the github issue. If you run into this breaking change, you can fix your pipeline by running an ALTER TABLE statement on your existing table.