How can I recover data from when my pipeline was broken?

1

My pipeline is pretty much exactly the “aws/secure” pipeline from the quickstart-examples.

For a period over the weekend, the pipeline was partially broken. The collector, enricher, S3 raw loader and S3 enriched loader were working. But the transformer and Redshift loader were not working.

Now the pipeline is fixed and everything is working, but there are a bunch of events in the S3 raw bucket and S3 enriched bucket that were never loaded into Redshift.

Is there a way to manually transform that data and load it into redshift?

2

If you are using the batch transformer on AWS you can set a sinceTimestamp for the transformer and then kick off a job manually - this will pick up on any data after this date and then send the required messages to the SQS loader to load the data.

3

@mike , is there a similar option for reprocess bad events to run them through kinesis stream/pipeline again? I had a schema version mismatch with iglu server and the tracker which sent all events to Bad_kinesis.

4

Yes - assuming that you have sunk those bad events to S3 you can use the Snowplow Event Recovery process to output events to a target Kinesis stream.