Failing to receive custom events in Snowplow Community Edition

For engineers Troubleshooting
1

Hi all -

We deployed Snowplow Community edition on GCP infrastructure and were successfully seeing both page view and custom events propagate to BigQuery. After changing the collector IP address and updating our load balancer to use new SSL certificates, we now only see the default page view events propagate to our warehouse. The custom events that were working previously are failing in validation. We see this message in our logs:

    "message": "[pool-1-thread-2] INFO org.http4s.client.middleware.Retry - Request threw an exception on attempt #1. Retrying after 100 milliseconds",
    "instance": {
      "name": "sp-enrich-tbck",
      "id": "3596669341240827270",
      "zone": "us-central1-b"
    }

We also see the custom events reaching the raw pubsub topic but failing enrichment:

1600×224 107 KB

We are not seeing any of the ‘bad data’ in the GCS bucket. Instead we’re just seeing an empty folder. We also see test events successfully propagate to BigQuery:

1220×854 110 KB

Thank you for taking the time to read this, and I appreciate any guidance or advice you all have!

1 Like