Can we use spectrum to query shredded data instead of enriched?

[dilyan]

Hi @rahul. You may also want to check out our follow up post on using Athena to query the shredded tables:

Using AWS Athena to query the shredded events

Recently, we looked into using AWS Athena to query the ‘good’ bucket on S3. The approach we outlined focussed on querying the ‘enriched’ unshredded data but we also wanted to see if we can query the shredded events directly from S3. Why use Athena? We’ve been using Athena to query the ‘bad’ data (ie data that fails validation) directly on S3. However, it also holds great promise for querying the ‘good’ Snowplow data in S3, potentially as an alternative to, or alongside, querying it in Redshift.…

Basically. you’ve identified the same key limitation as we did: the structure we are using to store the shredded data on S3 is not very friendly to the way Athena (and Redshift Spectrum) works.

I think your suggested approach is a sensible way of solving the issue. You may also find this post on the same topic interesting.

Let us know how you get on. We’d be keen to hear of your experiences with using Spectrum on shredded Snowplow data in S3.