Load with kafka to postgres

For engineers Kafka real-time pipeline
1

Hi,

I am trying to run the snowplow pipeline locally using Kafka and postgres.

According to the note mentioned on the this page snowplow only supports loader for Kinesis/PubSub.

Are there any plans to build a kafka loader by snowplow team? If anyone in the community knows any workaround to load data to postgres from kafka, it would really help.

Thanks

2

Hi @Gurankit_Pal_Singh,

At the moment, there is no loader compatible with Kafka, I’m afraid. So while we have a Kafka integration in the Collector and Enrich apps, we are not yet able to run the full pipeline outside of AWS/GCP.

For your specific use case (running Snowplow locally), we are actually working on adding loader functionality to Snowplow Micro. That way you would be able to run the whole pipeline in a single Docker container. Can’t say yet when this will be available, maybe 1–2 months from now.

We are also working on integrating the full pipeline with a cloud-agnostic messaging framework, which might or might not end up being Kafka. The goal here though is not running locally, but rather running in clouds other than AWS and GCP, using Kubernetes.