Snowplow Normalize DBT package can't find users_table macro

Try Snowplow
1

Hello,

I am working through a new snowplow set up and have used the snowplow-normalize dbt package to generate models for our self-describing events. I set up a packages.yml with the snowplow normalize package and installed it with dbt deps. I have successfully generated the models using the snowplow_normalize_model_gen.py script. However, whenever I try to run a dbt command I get the following error:

Compilation Error
  In dispatch: No macro named 'users_table' found within namespace: 'snowplow_normalize'
      Searched for: 'my_project.redshift__users_table', 'my_project.postgres__users_table', 'my_project.default__users_table', 'snowplow_normalize.redshift__users_table', 'snowplow_normalize.postgres__users_table', 'snowplow_normalize.default__users_table'

I see in the snowplow_normalize package macros directory there is a users_table macro and it is listed in the macros schema.yml. I didn’t configure any users table in the normalize config when I generated the models.

I’m a new dbt user so I might be missing something obvious. Any help is appreciated. Thanks

2

Ok, I should have read the docs more closely before getting too deep. snowplow-normalize doesn’t support redshift. :frowning:

3

That’s correct @Alex_Kudlick , because redshift already has the flat shredded tables we haven’t seen a need to add redshift support to the normalize package.

What is it you were hoping to do with the package as you may be able to build the models on top of our base macros to still take advantage of our incremental approach and other features.

4

Hi @Ryan thanks for the response.

redshift already has the flat shredded tables we haven’t seen a need to add redshift support to the normalize package.

Is this part of a quickstart setup? What generates those shredded tables? I didn’t exactly follow the the quickstart, what I have is our collector + enricher writing to kinesis, dumping to s3, then incremental load into redshift with airflow. I was hoping to automatically splat out our custom events via airflow jobs as well. Right now I am just writing sql to pull out each event type, which is not a huge deal since we don’t have that many custom event types.

5

The loader for redshift (and postgres) generated separate tables for the values in any self describing event or entity, rather than as columns in the main atomic events table like other warehouses. They won’t have the other atomic fields in those shredded tables. You can read more in the docs here https://docs.snowplow.io/docs/pipeline-components-and-applications/loaders-storage-targets/snowplow-rdb-loader/transforming-enriched-data/#shredded-data