Shredding & loading enriched events in near-real-time

[alex]

Hi @rgabo - it sounds like we are thinking about all this in the same way.

@rgabo

This gives us visibility into each step in the pipeline and also allows us to just use aws s3 mv and aws s3 rm --recursive where Sluice fails us.

Note that Sluice is no more - it was removed in Snowplow R91.

@rgabo

Is it fair to say that Dataflow Runner is a lightweight orchestrator (“runner”) for (not just) Snowplow pipelines?

Yes, that’s correct. Although we still use EmrEtlRunner internally for all core Snowplow pipelines, we use Dataflow Runner for our R&D and non-standard/non-Snowplow jobs on EMR.

Dataflow Runner is built around the Unix philosophy - all it does is run jobflows, currently on EMR only. You can schedule it any way you like. And it’s fully declarative - it’s just Avro, so you can generate, lint or visualise a dataflow anyway you like. (We are also planning a native integration between Factotum and Dataflow Runner in the future, so you get that “view-through” that you described between Airflow and EmrEtlRunner.)

Note that a future release of EmrEtlRunner will generate Dataflow Runner playbooks, and a later release will then remove the EMR orchestration functionality from EmrEtlRunner altogether:

Splitting EmrEtlRunner into snowplowctl and Dataflow Runner

As part of our drive to make the Snowplow community more collaborative and widen our network of open source contributors, we will be regularly posting our proposals for new features, apps and libraries under the Request for Comments section of our forum. You are welcome to post your own proposals too! This Request for Comments is to split our current EmrEtlRunner into two applications: snowplowctl, a command-line tool for working with Snowplow, and Dataflow Runner, a new open-source applicat…