We are please to announce the release of RDB Loader version 5.1.1
Bug fix for Databricks support with streaming transformer
This bug affects pipelines that use the streaming transformer to write parquet files for loading into Databricks. A bug was introduced in the 5.0.0 release that prevented the streaming transformer from writing to parquet. Affected users might have seen an error message like:
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
If you were affected by this bug, we hope you can now upgrade your loader and transformer to use version 5.1.1.
Upgrades to library dependencies
This release also upgrades several 3rd party library dependencies to newer versions. This is in line with our commitment to stay on top of potential security vulnerabilities that get reported over time.
Upgrading to 5.1.1
If you are already using a recent version of RDB Loader, then upgrading is as simple pulling the most recent docker images:
docker pull snowplow/transformer-kinesis:5.1.1
docker pull snowplow/transformer-pubsub:5.1.1
docker pull snowplow/rdb-loader-redshift:5.1.1
docker pull snowplow/rdb-loader-snowflake:5.1.1
docker pull snowplow/rdb-loader-databricks:5.1.1
The Snowplow docs site has a full guide to running and configuring the RDB Loader for different warehouse destinations.