Version 5.7.1 of Snowplow’s RDB loader is now released.
We added a new optional config field to the Spark batch transformer:
"skipSchemas": [ "iglu:com.example/my_schema_1/jsonschema/1-0-0", "iglu:com.example/my_schema_2/jsonschema/1-*-*", ]
If you supply the
skipSchemas config option, then the transformer’s output files will omit any columns using that schema. This works for any self-describing events and entities.
This feature could be helpful when recovering from edge-case schemas which for some reason cannot be loaded to the table. If you follow the usual rules of schema evolution then hopefully you will never need this feature.
We also made a few under-the-hood changes, including bumping some dependencies to newer versions, adding some tests, and removing some unhelpful logging lines.
If you are already using a recent version of RDB Loader then upgrading to
5.7.1 is as simple as pulling the newest docker images. There are no required changes to your config file.
docker pull snowplow/rdb-loader-redshift:5.7.1 docker pull snowplow/rdb-loader-snowflake:5.7.1 docker pull snowplow/rdb-loader-databricks:5.7.1 docker pull snowplow/transformer-pubsub:5.7.1 docker pull snowplow/transformer-kinesis:5.7.1 docker pull snowplow/transformer-kafka:5.7.1