Recover from EMR failures with deduplication?

[alex]

I know that @bernardosrulzon is aware - but for anybody else coming across this post, we have published a full tutorial on recovering from a pipeline failure involving cross-batch deduplication, here:

Recovering pipelines with cross-batch deduplication enabled [tutorial]

In Snowplow R86 Petra we released DynamoDB-powered cross-batch natural depulication, which effectively eliminated the problem of duplicates for the vast majority of Snowplow pipelines. A powerful new feature, this did also introduce some additional complexity around failed pipeline recovery, which Snowplow pipeline operators should be aware of. In this post we will explain how to safely recover a Snowplow pipeline with cross-batch deduplication enabled, introducing a new Spark app we have writt…