Service: Amazon S3; Status Code: 503; Error Code: 503 Slow Down issues

Today for the first time I started seeing the error below with S3 during the Archive Raw step.

Error: Slow Down (Service: Amazon S3; Status Code: 503; Error Code: 503 Slow Down; Request ID: A4DBBE2A70442C4D)

After waiting for over two hours to see if it’d recover, I killed the EMR cluster and plan to restart from that step.

Looking at AWS’s forums, I noticed that they ask to implement a backoff approach.

Is Snowplow able or have a plan to support such scenario?

UPDATE 1 It does feel like the jobs got lost - because the S3 bucket has no files left to move.

Error: java.lang.RuntimeException: Reducer task failed to copy 1061 files: s3://xxxxxxx-raw-events-xxxxxxxxxx/processing/2017-10-18-49577239542950406222078834590292547004774673200395059202-49577239542950406222078834703261829144483314152364834818.lzo etc

UPDATE 2 After restarting the emr job with options --skip staging,enrich,shred,elasticsearch, looking to restart from the S3 Raw Archive step, I got the failure below. The raw processing bucket has 0 files as mentioned above.

Exception in thread "main" java.lang.RuntimeException: Error running job at at at at at at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke( at sun.reflect.DelegatingMethodAccessorImpl.invoke( at java.lang.reflect.Method.invoke( at at org.apache.hadoop.util.RunJar.main( Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs:// at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus( at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus( at org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus( at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits( at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits( at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits( at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal( at org.apache.hadoop.mapreduce.Job$ at org.apache.hadoop.mapreduce.Job$ at Method) at at at org.apache.hadoop.mapreduce.Job.submit( at ... 10 more

UPDATE 3 Now I realized that I can’t run the RedShift and Postgres loader because I can’t start from that step without having any Raw file to archive.

Hey @cmartins - in this case, assuming that the raw events have been successfully archived, you can just start from the next step (rdb_load).

Someone ran into the same issue here.

I replied with a strategy to avoid this sort of thing.