Snowplow Loader - failing due to validation errors

For data modelers & consumers Snowflake
1

I’ve run all my historical events through the transformer successfully, but now it’s failing due to errors at the loading stage:

Error during enriched/run=2017-03-08-00-33-10/ load. Timestamp '16214-11-08T16:16:03.224Z' is not recognized
No new columns were added, safe to rerun.

And

Error during enriched/run=2017-03-31-00-33-10/ load. String 'CI6IwJvs_tICFe4Q0wodFJMOrg?org_url=[url removed]' is too long and would be truncated in 'VARCHAR(128)'
No new columns were added, safe to rerun.

I’ve marked the problem runs as ToSkip = True for now in DynamoDB, but I’d like to get them loaded in, and it looks like just a couple of rows causing the problems. Is there a way to tell the loader to ignore these errors?

Thanks!

Iain

2

Hi @iain. Right now you can only manually delete these particular rows from your archive (download file from S3, cut line, upload it back, restart pipeline). This will be addressed in next release.

3

Thanks Anton, I’ll do that!