EMR job failed in Hadoop Enrich step

For engineers AWS batch pipeline (Legacy)
1

Hello There,

(t9)    MOVE elasticbeanstalk-eu-west-1-347837873367/resources/environments/logs/publish/e-sdgsdjh23/i-sjdj3878/_var_log_tomcat8_rotated_localhost_access_log.txt1461222061.gz -> saltside-de-snowplow-market-name/logs/raw/processing/var_log_tomcat8_rotated_localhost_access_log.2016-04-21-07.eu-west-1.i-c3d4234f.txt.gz
      +-> saltside-de-snowplow-market-name/logs/raw/processing/var_log_tomcat8_rotated_localhost_access_log.2016-04-21-06.eu-west-1.i-c3d4234f.txt.gz
      +-> saltside-de-snowplow-market-name/logs/raw/processing/var_log_tomcat8_rotated_localhost_access_log.2016-04-21-07.eu-west-1.i-c3d4234f.txt.gz
      x elasticbeanstalk-eu-west-1-347837873367/resources/environments/logs/publish/e-uswjxqjezb/i-c3d4234f/_var_log_tomcat8_rotated_localhost_access_log.txt1461218462.gz
      x elasticbeanstalk-eu-west-1-347837873367/resources/environments/logs/publish/e-uswjxqjezb/i-c3d4234f/_var_log_tomcat8_rotated_localhost_access_log.txt1461222061.gz
F, [2016-04-21T08:12:28.499000 #23536] FATAL -- : 

Snowplow::EmrEtlRunner::EmrExecutionError (EMR jobflow j-8GLURHIDYRQ6 failed, check Amazon EMR console and Hadoop logs for details (help: https://github.com/snowplow/snowplow/wiki/Troubleshooting-jobs-on-Elastic-MapReduce). Data files not archived.
market-name-snowplow-emr-etl-runner: TERMINATING [STEP_FAILURE] ~ elapsed time n/a [2016-04-21 08:09:19 UTC - ]
 - 1. Elasticity Scalding Step: Enrich Raw Events: FAILED ~ 00:02:31 [2016-04-21 08:09:24 UTC - 2016-04-21 08:11:55 UTC]
 - 2. Elasticity S3DistCp Step: Shredded HDFS -> S3: CANCELLED ~ elapsed time n/a [ - ]
 - 3. Elasticity Scalding Step: Shred Enriched Events: CANCELLED ~ elapsed time n/a [ - ]
 - 4. Elasticity S3DistCp Step: Enriched HDFS _SUCCESS -> S3: CANCELLED ~ elapsed time n/a [ - ]
 - 5. Elasticity S3DistCp Step: Enriched HDFS -> S3: CANCELLED ~ elapsed time n/a [ - ]):
    /home/ec2-user/market/market-name/lib/snowplow-emr-etl-runner!/emr-etl-runner/lib/snowplow-emr-etl-runner/emr_job.rb:426:in `run'
    /home/ec2-user/market/market-name/lib/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/method_reference.rb:46:in `send_to'
    /home/ec2-user/market/market-name/lib/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts.rb:305:in `call_with'
    /home/ec2-user/market/market-name/lib/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/decorators.rb:159:in `common_method_added'
    /home/ec2-user/market/market-name/lib/snowplow-emr-etl-runner!/emr-etl-runner/lib/snowplow-emr-etl-runner/runner.rb:68:in `run'
    /home/ec2-user/market/market-name/lib/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/method_reference.rb:46:in `send_to'
    /home/ec2-user/market/market-name/lib/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts.rb:305:in `call_with'
2

Hello @birju1100

Seems something went wrong with your enrichment process. Can you show us your EMR logs?
Logs can be found at AWS Console: EMR -> Your job -> Steps -> Elasticity Scalding Step: Enrich Raw Events -> stderr.
Could you also provide a command with which you started EmrEtlRunner?

Cheers,
Anton

3

Hey Anton,

I have re run with --skip staging it working now.