We are very happy to announce the release of the snowplow-web v.0.9.2 dbt package.
This release optimises performance in Databricks for incremental models in the derived schemas. In addition it clarifies some documentation around how to set up your dbt packages to work optimally with Databricks depending on whether or not you have the Unity Catalog enabled and which dbt connector you are using.
Features
- Optimize performance in Databricks for incremental models (Close #113)
Upgrading
To upgrade simply bump the snowplow-web version in your packages.yml
file. If you want to take advantage of the performance optimisations in already existing incremental tables in your Databricks environment, run the following commands against Databricks for all of the incremental tables in your derived schema:
ALTER TABLE {derived_schema}.{table_name} SET TBLPROPERTIES (delta.autoOptimize.optimizeWrite = true, delta.autoOptimize.autoCompact = true);
By default, these are the snowplow_web_page_views
, snowplow_web_sessions
, and snowplow_web_users
tables in the derived schema.