How can we Reload the missing data in to redshift

For engineers AWS real-time pipeline
1

I have set up my Snowplow collector, enricher, transformer, and RDB loader using ECS. It’s been running smoothly for the past 6 months. However, in the last two days, I haven’t received any events in the atomic.events table. Upon checking the logs, we identified an issue from 2024-02-16 20:00 to 2024-02-19 00:00.

I’ve thoroughly examined the collector and enricher, but there seem to be no issues, and the data has been stored in S3 for all hours over the past three days. From ECS, I traced the problem to the transformer service, which was overloaded and hanging. Consequently, I updated the service and added one more desired task vCPU. Within 5 minutes, the data started loading into the RDB loader.

However, the data was only restored after 2024-02-18. I lost the data from 2024-02-16 (half-day data) to 2024-02-17 (full data).

Now, the question is, how can I restore the lost data? Fortunately, I have all the data in the enricher’s good events S3 bucket.

Here are the versions of the Snowplow components:

  • snowplow/scala-stream-collector-kinesis:2.9.0
  • snowplow/snowplow-enrich-kinesis:3.8.0
  • snowplow/transformer-kinesis:5.6.0
  • snowplow/rdb-loader-redshift:5.6.0
1 Like