We are pleased to release version 0.6.0 of Snowplow S3 Loader.
In this release, we have renamed the project from Kinesis S3 to S3 Loader since NSQ support is added.
To know more about other improvements and added features you can read the blog post.
2 Likes
Nice work @aldemirenes!
What’s this going to be called when it adds functionality for other sinks (e.g., blob storage, cloud storage etc)?
Hey @mike - good question!
First we need to figure out if it makes sense to house all of the blob storage sinks in the same loader codebase (versus e.g. extracting shared functionality into common libraries).
More and more capabilities in a single loader means a much more challenging integration test matrix (e.g. Google Cloud Pub/Sub -> Azure Data Lake Store).
Hi,
I am trying to move from the snowplow batch to streaming pipeline, and have added a couple of PRs to the s3-loader, which I would like to see released. (I am trying to avoid building my own branch for use in production).
How soon do you think will we see the next release of s3 loader?
Hi, currently we are running Snowplow pipeline with ASW Kinesis as a message queue. One of the steps of our pipeline is persisting enriched events from Kinesis to S3 with Snowplow S3 Loader. We want to migrate from Kinesis to Kafka and and cannot find the replacement for that step since S3 Loader doesn’t support Kafka source.
One possible option is to use Kafka S3 Connector, but then we break the subsequent steps of the pipeline.
Do you have on the roadmap supporting Kafka source for Snowplow S3 Loader and is there any timeline on that?
Hi @jahstreet,
No plans to support a Kafka S3 loader in the near future. What’s more, if you’re loading to a database and want to switch to Kafka, you should know we only support loaders that consume from Kinesis at the moment. (or s3 in some cases, but these are planned to move to stream-only too).
There are others who are successfully running the Kafka pipeline, so perhaps it’s worth taking a look at previous threads on the topic or opening one to discuss it.
Best,
Thx @Colm, will continue the searching.
Hi @Colm, kafka stream collector + kafka stream enrich + ? (storage loader).
not able found any documentation regarding storage loader after kafka enrichment.
Regards,
Vipin
Hi @Vipin,
What’s more, if you’re loading to a database and want to switch to Kafka, you should know we only support loaders that consume from Kinesis at the moment. (or s3 in some cases, but these are planned to move to stream-only too).
There are no supported database loaders from Kafka. Others who have achieved this have done so by building it themselves - my message was a suggestion to seek out those who have done so if one is interested in doing the same.
Best,