Multiple dbt packages upgraded, with more on the way!

1

Summary

We’ve just released new versions of our utils (0.14.0), web (0.14.0), mobile (0.7.0), and normalize (0.3.0) dbt packages, and we expect to release a new version of media player (0.5.0) tomorrow (29th March) with ecommerce (0.4.0) and fractribution (0.3.0) being released in the coming days (links will be added once released).

While these releases contain a few additional fixes and some new features, the core changes in the new versions are:

  1. Replace the snowplow_incremental materialization with an optimized version of the standard incremental one, which will require at least version 1.4 of dbt. To make use of this you must add the following to your dbt_project.yml file and read here for any custom models
    # dbt_project.yml
...
dispatch:
  - macro_namespace: dbt
    search_order: ['snowplow_utils', 'dbt']
  2. We have also made changes to the deduplication logic for redshift+postgres (where supported) to align with the other warehouses i.e. we no longer throw away all instances of a duplicate but instead keep one at random. In the case of joining contexts or self describing events this means you must now deduplicate your sde/context table BEFORE joining to any model tables. If you use a different warehouse, or don’t have any custom models, then there is no changes required from you for this.

Check out our migration guides for further details about any changes you need to make to upgrade your versions.

Why these changes, why now?

We have made the decision to make these two somewhat disruptive changes at the same time to minimise the frequency with which you have to edit your models. We also made these changes so that we can more easily update our own packages and models in the future, as they are now more standardised across the warehouses, and to take advantage of some of the newer features of dbt such as the on_schema_change config and the incremental predicates released in verison 1.4 (without which we couldn’t move away from our custom materialization).

In general these changes, and some planned changes for later this year, come down to reducing our tech debt and streamlining the package internals so we can more quickly respond to new feature requests and reduce the risk of bugs. We have come a long way since releasing our first dbt package nearly 2 years ago, which itself was based off an older sql-runner model, so it is the right time to take all that learning and tackle some of the core functionality of our packages that has remained unchanged since the start. As we accelerate our development of the packages, there may be more large changes still, but these all get us closer to a stable 1.0 release and will continue to add value to users.

4 Likes