Snowplow Snowflake Loader 0.8.0 released

Announcements New releases
1

We’re happy to announce version 0.8.0 of Snowplow Snowflake Loader.

This is the first release working on top of Apache Spark 3 and AWS EMR 6. Apart from this major upgrade it also brings several improvements and bug fixes. Here are most notable ones:

  • New --debug flag provides additional output from Loader, such as all SQL statements being executed and current state of events table (#120)
  • All VARCHAR columns are properly truncated, preventing loading failures (#126)
  • Timeout exception thrown be Transformer scanning very big manifests has been fixed (#124)

The upgrade guide is available on our docs site.

4 Likes
2

Hey @anton!

Thanks for hard work and this new release!

I recently upgraded to 0.8.0 and that was a smooth upgrade, however the link to the upgrade guide is not pointing to any valid section anymore.

As I am in the process of migrated our Snowflake stages to storage integration, I wanted to double check the process there, but unfortunately I can’t now :frowning:

Is there a plan to restore the documentation page?
Also with the latest releases I guess the compatibility matrix could be updated :wink:

Thanks again for the hard work!

3

Hey @AcidFlow,

I think there might have something else happened between day you posted and today, but I see the storage integration guide being on our docs site: https://docs.snowplowanalytics.com/docs/open-source-components-and-applications/snowplow-snowflake-loader/upgrade-guide/#upgrading-from-0-6-x-to-0-7-x (it was added in 0.7.0). Or do you mean the wiki page? Unfortunately, I don’t think it will be live again as we’re migrating from Github wiki.

On the Compatibility Matrix - yup, going to update it, thanks for spotting.

4

Oh sorry, my bad regarding the storage integration. You’re totally right support was added in 0.7.0.

However I remember seeing something about updating the cluster configuration to use AMI version 6.1.0 and to remove some custom configuration in the playbook :

"--conf", "spark.hadoop.mapreduce.job.outputformat.class=com.snowplowanalytics.snowflake.transformer.S3OutputFormat"
1 Like
5

This helped me solved my problem thanks a lot