EMR intermittently fails at Loading S3 to Redshift

Hoping to move to stream in the near future.

That’s a very good idea anyway as we’re deprecating Spark Enrich. Have a read Paul’s upgrading guide: AWS batch pipeline to real-time pipeline upgrade guide.

One more thing I forgot to mention is consolidate_shredded_output setting from R112, which also helped to increase stability of loading step.

1 Like