Snowplow data modeling issues when updating schema

For data modelers & consumers SQL Runner
1

Hi Snowplowers,

I am currently working on the Snowplow data modeling pipeline using SQL runner. I met a problem. Each time we add or update a self-describing schema, then the data modeling pipeline would fail since the target table doesn’t match the source table anymore. For now, I will delete the whole derived and scratch dataset and then create new ones manually. But I found the operation is quite frequent and dull. So I am wondering whether you have any good idea about this problem. Thanks!

2

Hi @phxtorise,

Repeatedly deleting your datasets does sounds like a pain! I can try and help.

Are you able to give a bit more detail on the problem you are facing? What derived and scratch datasets are you referring to? Are you using the web v1 model or have you created your own data model?

Thanks,

Will