Trouble Configuring Snowplow Pipeline with AWS S3 as the Collector

For engineers
1

I’m currently facing difficulties setting up Snowplow with AWS S3 as the collector and would appreciate some assistance.

Here’s a brief overview of what I’ve done so far:

  1. I’ve created an AWS S3 bucket to store the raw event data.
  2. I’ve set up the Snowplow collector using the Snowplow Mini Collector Stack template and configured it to send events to the S3 bucket.
  3. I’ve adjusted the necessary configuration files (e.g., config.hocon, iglu_resolver.json, etc.) to reflect my setup.

However, when I attempt to send events to the collector endpoint, I’m not seeing any data being stored in the S3 bucket. I’ve checked the collector logs, and there don’t seem to be any obvious errors.

Could someone please provide some guidance on troubleshooting this issue? Are there any specific logs or configurations I should be checking to identify the source of the problem? Any help or suggestions would be greatly appreciated. :slightly_smiling_face: :melting_face: Thank you!

2

Hi Joe,

This looks like potentially a bit of an unusual setup (as Snowplow Mini does not typically send any data to S3, rather it sends to Elasticsearch. Are you able to point me to the docs / guide you are following? (If you aren’t using quick start that may be the better option).