We have an ETL that hangs for about 3 weeks on Enrich stage. Usually takes about a minute for the whole cluster to run.
controller log of the cluster step:
2018-10-04T04:36:29.999Z INFO Ensure step 3 jar file command-runner.jar
2018-10-04T04:36:29.999Z INFO StepRunner: Created Runner for step 3
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar spark-submit --class com.snowplowanalytics.snowplow.enrich.spark.EnrichJob --master yarn --deploy-mode cluster s3://snowplow-hosted-assets-us-west-2/3-enrich/spark-enrich/snowplow-spark-enrich-1.14.0.jar --input-format clj-tomcat --etl-timestamp 1538627422352 --iglu-config ewogICJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3MuaWdsdS9yZXNvbHZlci1jb25maWcvanNvbnNjaGVtYS8xLTAtMSIsCiAgImRhdGEiOiB7CiAgICAiY2FjaGVTaXplIjogNTAwLAogICAgInJlcG9zaXRvcmllcyI6IFsKICAgICAgewogICAgICAgICJuYW1lIjogIklnbHUgQ2VudHJhbCIsCiAgICAgICAgInByaW9yaXR5IjogMCwKICAgICAgICAidmVuZG9yUHJlZml4ZXMiOiBbICJjb20uc25vd3Bsb3dhbmFseXRpY3MiIF0sCiAgICAgICAgImNvbm5lY3Rpb24iOiB7CiAgICAgICAgICAiaHR0cCI6IHsKICAgICAgICAgICAgInVyaSI6ICJodHRwOi8vaWdsdWNlbnRyYWwuY29tIgogICAgICAgICAgfQogICAgICAgIH0KICAgICAgfQogICAgXQogIH0KfQo= --enrichments eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9lbnJpY2htZW50cy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6W3sic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvcmVmZXJlcl9wYXJzZXIvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsibmFtZSI6InJlZmVyZXJfcGFyc2VyIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJpbnRlcm5hbERvbWFpbnMiOltdfX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvdWFfcGFyc2VyX2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3ciLCJuYW1lIjoidWFfcGFyc2VyX2NvbmZpZyIsImVuYWJsZWQiOnRydWUsInBhcmFtZXRlcnMiOnt9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9hbm9uX2lwL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7Im5hbWUiOiJhbm9uX2lwIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJhbm9uT2N0ZXRzIjoxfX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvaXBfbG9va3Vwcy9qc29uc2NoZW1hLzItMC0wIiwiZGF0YSI6eyJuYW1lIjoiaXBfbG9va3VwcyIsInZlbmRvciI6ImNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdyIsImVuYWJsZWQiOnRydWUsInBhcmFtZXRlcnMiOnsiZ2VvIjp7ImRhdGFiYXNlIjoiR2VvTGl0ZTItQ2l0eS5tbWRiIiwidXJpIjoiaHR0cDovL3Nub3dwbG93LWhvc3RlZC1hc3NldHMuczMuYW1hem9uYXdzLmNvbS90aGlyZC1wYXJ0eS9tYXhtaW5kIn19fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jdXJyZW5jeV9jb252ZXJzaW9uX2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJlbmFibGVkIjpmYWxzZSwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwibmFtZSI6ImN1cnJlbmN5X2NvbnZlcnNpb25fY29uZmlnIiwicGFyYW1ldGVycyI6eyJhY2NvdW50VHlwZSI6IkRFVkVMT1BFUiIsImFwaUtleSI6Int7S0VZfX0iLCJiYXNlQ3VycmVuY3kiOiJVU0QiLCJyYXRlQXQiOiJFT0RfUFJJT1IifX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cuZW5yaWNobWVudHMvc3FsX3F1ZXJ5X2VucmljaG1lbnRfY29uZmlnL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InZlbmRvciI6ImNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cyIsIm5hbWUiOiJzcWxfcXVlcnlfZW5yaWNobWVudF9jb25maWciLCJlbmFibGVkIjpmYWxzZSwicGFyYW1ldGVycyI6eyJpbnB1dHMiOlt7InBsYWNlaG9sZGVyIjoxLCJwb2pvIjp7ImZpZWxkIjoidXNlcl9pZCJ9fSx7InBsYWNlaG9sZGVyIjoxLCJqc29uIjp7ImZpZWxkIjoiY29udGV4dHMiLCJzY2hlbWFDcml0ZXJpb24iOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jbGllbnRfc2Vzc2lvbi9qc29uc2NoZW1hLzEtKi0qIiwianNvblBhdGgiOiIkLnVzZXJJZCJ9fSx7InBsYWNlaG9sZGVyIjoyLCJwb2pvIjp7ImZpZWxkIjoiYXBwX2lkIn19XSwiZGF0YWJhc2UiOnsicG9zdGdyZXNxbCI6eyJob3N0IjoiY2x1c3RlcjAxLnJlZHNoaWZ0LmFjbWUuY29tIiwicG9ydCI6NTQzOSwic3NsTW9kZSI6dHJ1ZSwidXNlcm5hbWUiOiJzbm93cGxvd19lbnJpY2hfcm8iLCJwYXNzd29yZCI6IjFhc0lrSmVkIiwiZGF0YWJhc2UiOiJjcm0ifX0sInF1ZXJ5Ijp7InNxbCI6IlNFTEVDVCB1c2VybmFtZSwgZW1haWxfYWRkcmVzcywgZGF0ZV9vZl9iaXJ0aCBGUk9NIHRibF91c2VycyBXSEVSRSB1c2VyID0gPyBBTkQgY2xpZW50ID0gPyBMSU1JVCAxIn0sIm91dHB1dCI6eyJleHBlY3RlZFJvd3MiOiJBVF9NT1NUX09ORSIsImpzb24iOnsic2NoZW1hIjoiaWdsdTpjb20uYWNtZS91c2VyL2pzb25zY2hlbWEvMS0wLTAiLCJkZXNjcmliZXMiOiJBTExfUk9XUyIsInByb3BlcnR5TmFtZXMiOiJDQU1FTF9DQVNFIn19LCJjYWNoZSI6eyJzaXplIjozMDAwLCJ0dGwiOjYwfX19fSx7InNjaGVtYSI6ImlnbHU6Y29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93L2V2ZW50X2ZpbmdlcnByaW50X2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3ciLCJuYW1lIjoiZXZlbnRfZmluZ2VycHJpbnRfY29uZmlnIiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJleGNsdWRlUGFyYW1ldGVycyI6WyJlaWQiLCJudWlkIiwic3RtIiwiY3YiXSwiaGFzaEFsZ29yaXRobSI6Ik1ENSJ9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cy9odHRwX2hlYWRlcl9leHRyYWN0b3JfY29uZmlnL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InZlbmRvciI6ImNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cyIsIm5hbWUiOiJodHRwX2hlYWRlcl9leHRyYWN0b3JfY29uZmlnIiwiZW5hYmxlZCI6ZmFsc2UsInBhcmFtZXRlcnMiOnsiaGVhZGVyc1BhdHRlcm4iOiIuKiJ9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cy93ZWF0aGVyX2VucmljaG1lbnRfY29uZmlnL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7ImVuYWJsZWQiOmZhbHNlLCJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cuZW5yaWNobWVudHMiLCJuYW1lIjoid2VhdGhlcl9lbnJpY2htZW50X2NvbmZpZyIsInBhcmFtZXRlcnMiOnsiYXBpS2V5Ijoie3tLRVl9fSIsImNhY2hlU2l6ZSI6NTEwMCwiZ2VvUHJlY2lzaW9uIjoxLCJhcGlIb3N0IjoiaGlzdG9yeS5vcGVud2VhdGhlcm1hcC5vcmciLCJ0aW1lb3V0Ijo1fX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvY29va2llX2V4dHJhY3Rvcl9jb25maWcvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsibmFtZSI6ImNvb2tpZV9leHRyYWN0b3JfY29uZmlnIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJjb29raWVzIjpbInNwIl19fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9qYXZhc2NyaXB0X3NjcmlwdF9jb25maWcvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwibmFtZSI6ImphdmFzY3JpcHRfc2NyaXB0X2NvbmZpZyIsImVuYWJsZWQiOmZhbHNlLCJwYXJhbWV0ZXJzIjp7InNjcmlwdCI6IlpuVnVZM1JwYjI0Z2NISnZZMlZ6Y3lobGRtVnVkQ2tnZXcwS0RRb2dJSFpoY2lCd2JHRjBabTl5YlNBOUlHVjJaVzUwTG1kbGRGQnNZWFJtYjNKdEtDa3NEUW9nSUNBZ0lDQmhjSEJKWkNBZ0lDQTlJR1YyWlc1MExtZGxkRUZ3Y0Y5cFpDZ3BPdzBLRFFvZ0lHbG1JQ2h3YkdGMFptOXliU0E5UFNBaWMyVnlkbVZ5SWlBbUppQmhjSEJKWkNBaFBTQWljMlZqY21WMElpa2dldzBLSUNBZ0lIUm9jbTkzSUNKVFpYSjJaWEl0YzJsa1pTQmxkbVZ1ZENCb1lYTWdhVzUyWVd4cFpDQmhjSEJmYVdRNklDSWdLeUJoY0hCSlpEc05DaUFnZlEwS0lDQU5DaUFnYVdZZ0tHRndjRWxrSUQwOUlHNTFiR3dwSUhzTkNpQWdJQ0J5WlhSMWNtNGdXMTA3RFFvZ0lIME5DZzBLSUNBdkx5QlZjMlVnYm1WM0lGTjBjbWx1WnlncElHSmxZMkYxYzJVZ2FIUjBjRG92TDI1bGJITnZibmRsYkd4ekxtNWxkQzh5TURFeUx6QXlMMnB6YjI0dGMzUnlhVzVuYVdaNUxYZHBkR2d0YldGd2NHVmtMWFpoY21saFlteGxjeThOQ2lBZ2RtRnlJR0Z3Y0Vsa1ZYQndaWElnUFNCdVpYY2dVM1J5YVc1bktHRndjRWxrTG5SdlZYQndaWEpEWVhObEtDa3BPdzBLRFFvZ0lISmxkSFZ5YmlCYklIc2djMk5vWlcxaE9pQWlhV2RzZFRwamIyMHVZV050WlM5bWIyOHZhbk52Ym5OamFHVnRZUzh4TFRBdE1DSXNEUW9nSUNBZ0lDQWdJQ0FnSUNBZ0lDQmtZWFJoT2lCN0lHRndjRWxrVlhCd1pYSTZJR0Z3Y0Vsa1ZYQndaWElnZlEwS0lDQWdJQ0FnSUNBZ0lDQjlJRjA3RFFwOSJ9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91c2VyX2FnZW50X3V0aWxzX2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3ciLCJuYW1lIjoidXNlcl9hZ2VudF91dGlsc19jb25maWciLCJlbmFibGVkIjp0cnVlLCJwYXJhbWV0ZXJzIjp7fX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvY2FtcGFpZ25fYXR0cmlidXRpb24vanNvbnNjaGVtYS8xLTAtMSIsImRhdGEiOnsibmFtZSI6ImNhbXBhaWduX2F0dHJpYnV0aW9uIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJtYXBwaW5nIjoic3RhdGljIiwiZmllbGRzIjp7Im1rdE1lZGl1bSI6WyJ1dG1fbWVkaXVtIl0sIm1rdFNvdXJjZSI6WyJ1dG1fc291cmNlIl0sIm1rdFRlcm0iOlsidXRtX3Rlcm0iXSwibWt0Q29udGVudCI6WyJ1dG1fY29udGVudCJdLCJta3RDYW1wYWlnbiI6WyJ1dG1fY2FtcGFpZ24iXX19fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cy9hcGlfcmVxdWVzdF9lbnJpY2htZW50X2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cuZW5yaWNobWVudHMiLCJuYW1lIjoiYXBpX3JlcXVlc3RfZW5yaWNobWVudF9jb25maWciLCJlbmFibGVkIjpmYWxzZSwicGFyYW1ldGVycyI6eyJpbnB1dHMiOlt7ImtleSI6InVzZXIiLCJwb2pvIjp7ImZpZWxkIjoidXNlcl9pZCJ9fSx7ImtleSI6InVzZXIiLCJqc29uIjp7ImZpZWxkIjoiY29udGV4dHMiLCJzY2hlbWFDcml0ZXJpb24iOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jbGllbnRfc2Vzc2lvbi9qc29uc2NoZW1hLzEtKi0qIiwianNvblBhdGgiOiIkLnVzZXJJZCJ9fSx7ImtleSI6ImNsaWVudCIsInBvam8iOnsiZmllbGQiOiJhcHBfaWQifX1dLCJhcGkiOnsiaHR0cCI6eyJtZXRob2QiOiJHRVQiLCJ1cmkiOiJodHRwOi8vYXBpLmFjbWUuY29tL3VzZXJzL3t7Y2xpZW50fX0ve3t1c2VyfX0/Zm9ybWF0PWpzb24iLCJ0aW1lb3V0IjoyMDAwLCJhdXRoZW50aWNhdGlvbiI6eyJodHRwQmFzaWMiOnsidXNlcm5hbWUiOiJ4eHgiLCJwYXNzd29yZCI6Inl5eSJ9fX19LCJvdXRwdXRzIjpbeyJzY2hlbWEiOiJpZ2x1OmNvbS5hY21lL3VzZXIvanNvbnNjaGVtYS8xLTAtMCIsImpzb24iOnsianNvblBhdGgiOiIkLnJlY29yZCJ9fV0sImNhY2hlIjp7InNpemUiOjMwMDAsInR0bCI6NjB9fX19XX0= --input-folder hdfs:///local/snowplow/raw-events/* --output-folder hdfs:///local/snowplow/enriched-events/ --bad-folder s3://infinigrow-logs/etlrunner/enriched/bad/run=2018-10-04-04-30-22/'
> INFO Environment:
> PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/opt/aws/bin
> LESS_TERMCAP_md=[01;38;5;208m
> LESS_TERMCAP_me=[0m
> HISTCONTROL=ignoredups
> LESS_TERMCAP_mb=[01;31m
> AWS_AUTO_SCALING_HOME=/opt/aws/apitools/as
> UPSTART_JOB=rc
> LESS_TERMCAP_se=[0m
> HISTSIZE=1000
> HADOOP_ROOT_LOGGER=INFO,DRFA
> JAVA_HOME=/etc/alternatives/jre
> AWS_DEFAULT_REGION=us-west-2
> AWS_ELB_HOME=/opt/aws/apitools/elb
> LESS_TERMCAP_us=[04;38;5;111m
> EC2_HOME=/opt/aws/apitools/ec2
> TERM=linux
> XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt
> runlevel=3
> LANG=en_US.UTF-8
> AWS_CLOUDWATCH_HOME=/opt/aws/apitools/mon
> MAIL=/var/spool/mail/hadoop
> LESS_TERMCAP_ue=[0m
> LOGNAME=hadoop
> PWD=/
> LANGSH_SOURCED=1
> HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-14Q6W9V3K08FF/tmp
> _=/etc/alternatives/jre/bin/java
> CONSOLETYPE=serial
> RUNLEVEL=3
> LESSOPEN=||/usr/bin/lesspipe.sh %s
> previous=N
> UPSTART_EVENTS=runlevel
> AWS_PATH=/opt/aws
> USER=hadoop
> UPSTART_INSTANCE=
> PREVLEVEL=N
> HADOOP_LOGFILE=syslog
> PYTHON_INSTALL_LAYOUT=amzn
> HOSTNAME=ip-172-31-18-60
> NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat
> HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-14Q6W9V3K08FF
> EC2_AMITOOL_HOME=/opt/aws/amitools/ec2
> SHLVL=5
> HOME=/home/hadoop
> HADOOP_IDENT_STRING=hadoop
> INFO redirectOutput to /mnt/var/log/hadoop/steps/s-14Q6W9V3K08FF/stdout
> INFO redirectError to /mnt/var/log/hadoop/steps/s-14Q6W9V3K08FF/stderr
> INFO Working dir /mnt/var/lib/hadoop/steps/s-14Q6W9V3K08FF
> INFO ProcessRunner started child process 8876 :
> hadoop 8876 4000 0 04:36 ? 00:00:00 bash /usr/lib/hadoop/bin/hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar spark-submit --class com.snowplowanalytics.snowplow.enrich.spark.EnrichJob --master yarn --deploy-mode cluster s3://snowplow-hosted-assets-us-west-2/3-enrich/spark-enrich/snowplow-spark-enrich-1.14.0.jar --input-format clj-tomcat --etl-timestamp 1538627422352 --iglu-config ewogICJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3MuaWdsdS9yZXNvbHZlci1jb25maWcvanNvbnNjaGVtYS8xLTAtMSIsCiAgImRhdGEiOiB7CiAgICAiY2FjaGVTaXplIjogNTAwLAogICAgInJlcG9zaXRvcmllcyI6IFsKICAgICAgewogICAgICAgICJuYW1lIjogIklnbHUgQ2VudHJhbCIsCiAgICAgICAgInByaW9yaXR5IjogMCwKICAgICAgICAidmVuZG9yUHJlZml4ZXMiOiBbICJjb20uc25vd3Bsb3dhbmFseXRpY3MiIF0sCiAgICAgICAgImNvbm5lY3Rpb24iOiB7CiAgICAgICAgICAiaHR0cCI6IHsKICAgICAgICAgICAgInVyaSI6ICJodHRwOi8vaWdsdWNlbnRyYWwuY29tIgogICAgICAgICAgfQogICAgICAgIH0KICAgICAgfQogICAgXQogIH0KfQo= --enrichments eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9lbnJpY2htZW50cy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6W3sic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvcmVmZXJlcl9wYXJzZXIvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsibmFtZSI6InJlZmVyZXJfcGFyc2VyIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJpbnRlcm5hbERvbWFpbnMiOltdfX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvdWFfcGFyc2VyX2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3ciLCJuYW1lIjoidWFfcGFyc2VyX2NvbmZpZyIsImVuYWJsZWQiOnRydWUsInBhcmFtZXRlcnMiOnt9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9hbm9uX2lwL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7Im5hbWUiOiJhbm9uX2lwIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJhbm9uT2N0ZXRzIjoxfX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvaXBfbG9va3Vwcy9qc29uc2NoZW1hLzItMC0wIiwiZGF0YSI6eyJuYW1lIjoiaXBfbG9va3VwcyIsInZlbmRvciI6ImNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdyIsImVuYWJsZWQiOnRydWUsInBhcmFtZXRlcnMiOnsiZ2VvIjp7ImRhdGFiYXNlIjoiR2VvTGl0ZTItQ2l0eS5tbWRiIiwidXJpIjoiaHR0cDovL3Nub3dwbG93LWhvc3RlZC1hc3NldHMuczMuYW1hem9uYXdzLmNvbS90aGlyZC1wYXJ0eS9tYXhtaW5kIn19fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jdXJyZW5jeV9jb252ZXJzaW9uX2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJlbmFibGVkIjpmYWxzZSwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwibmFtZSI6ImN1cnJlbmN5X2NvbnZlcnNpb25fY29uZmlnIiwicGFyYW1ldGVycyI6eyJhY2NvdW50VHlwZSI6IkRFVkVMT1BFUiIsImFwaUtleSI6Int7S0VZfX0iLCJiYXNlQ3VycmVuY3kiOiJVU0QiLCJyYXRlQXQiOiJFT0RfUFJJT1IifX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cuZW5yaWNobWVudHMvc3FsX3F1ZXJ5X2VucmljaG1lbnRfY29uZmlnL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InZlbmRvciI6ImNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cyIsIm5hbWUiOiJzcWxfcXVlcnlfZW5yaWNobWVudF9jb25maWciLCJlbmFibGVkIjpmYWxzZSwicGFyYW1ldGVycyI6eyJpbnB1dHMiOlt7InBsYWNlaG9sZGVyIjoxLCJwb2pvIjp7ImZpZWxkIjoidXNlcl9pZCJ9fSx7InBsYWNlaG9sZGVyIjoxLCJqc29uIjp7ImZpZWxkIjoiY29udGV4dHMiLCJzY2hlbWFDcml0ZXJpb24iOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jbGllbnRfc2Vzc2lvbi9qc29uc2NoZW1hLzEtKi0qIiwianNvblBhdGgiOiIkLnVzZXJJZCJ9fSx7InBsYWNlaG9sZGVyIjoyLCJwb2pvIjp7ImZpZWxkIjoiYXBwX2lkIn19XSwiZGF0YWJhc2UiOnsicG9zdGdyZXNxbCI6eyJob3N0IjoiY2x1c3RlcjAxLnJlZHNoaWZ0LmFjbWUuY29tIiwicG9ydCI6NTQzOSwic3NsTW9kZSI6dHJ1ZSwidXNlcm5hbWUiOiJzbm93cGxvd19lbnJpY2hfcm8iLCJwYXNzd29yZCI6IjFhc0lrSmVkIiwiZGF0YWJhc2UiOiJjcm0ifX0sInF1ZXJ5Ijp7InNxbCI6IlNFTEVDVCB1c2VybmFtZSwgZW1haWxfYWRkcmVzcywgZGF0ZV9vZl9iaXJ0aCBGUk9NIHRibF91c2VycyBXSEVSRSB1c2VyID0gPyBBTkQgY2xpZW50ID0gPyBMSU1JVCAxIn0sIm91dHB1dCI6eyJleHBlY3RlZFJvd3MiOiJBVF9NT1NUX09ORSIsImpzb24iOnsic2NoZW1hIjoiaWdsdTpjb20uYWNtZS91c2VyL2pzb25zY2hlbWEvMS0wLTAiLCJkZXNjcmliZXMiOiJBTExfUk9XUyIsInByb3BlcnR5TmFtZXMiOiJDQU1FTF9DQVNFIn19LCJjYWNoZSI6eyJzaXplIjozMDAwLCJ0dGwiOjYwfX19fSx7InNjaGVtYSI6ImlnbHU6Y29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93L2V2ZW50X2ZpbmdlcnByaW50X2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3ciLCJuYW1lIjoiZXZlbnRfZmluZ2VycHJpbnRfY29uZmlnIiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJleGNsdWRlUGFyYW1ldGVycyI6WyJlaWQiLCJudWlkIiwic3RtIiwiY3YiXSwiaGFzaEFsZ29yaXRobSI6Ik1ENSJ9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cy9odHRwX2hlYWRlcl9leHRyYWN0b3JfY29uZmlnL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InZlbmRvciI6ImNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cyIsIm5hbWUiOiJodHRwX2hlYWRlcl9leHRyYWN0b3JfY29uZmlnIiwiZW5hYmxlZCI6ZmFsc2UsInBhcmFtZXRlcnMiOnsiaGVhZGVyc1BhdHRlcm4iOiIuKiJ9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cy93ZWF0aGVyX2VucmljaG1lbnRfY29uZmlnL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7ImVuYWJsZWQiOmZhbHNlLCJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cuZW5yaWNobWVudHMiLCJuYW1lIjoid2VhdGhlcl9lbnJpY2htZW50X2NvbmZpZyIsInBhcmFtZXRlcnMiOnsiYXBpS2V5Ijoie3tLRVl9fSIsImNhY2hlU2l6ZSI6NTEwMCwiZ2VvUHJlY2lzaW9uIjoxLCJhcGlIb3N0IjoiaGlzdG9yeS5vcGVud2VhdGhlcm1hcC5vcmciLCJ0aW1lb3V0Ijo1fX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvY29va2llX2V4dHJhY3Rvcl9jb25maWcvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsibmFtZSI6ImNvb2tpZV9leHRyYWN0b3JfY29uZmlnIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJjb29raWVzIjpbInNwIl19fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9qYXZhc2NyaXB0X3NjcmlwdF9jb25maWcvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwibmFtZSI6ImphdmFzY3JpcHRfc2NyaXB0X2NvbmZpZyIsImVuYWJsZWQiOmZhbHNlLCJwYXJhbWV0ZXJzIjp7InNjcmlwdCI6IlpuVnVZM1JwYjI0Z2NISnZZMlZ6Y3lobGRtVnVkQ2tnZXcwS0RRb2dJSFpoY2lCd2JHRjBabTl5YlNBOUlHVjJaVzUwTG1kbGRGQnNZWFJtYjNKdEtDa3NEUW9nSUNBZ0lDQmhjSEJKWkNBZ0lDQTlJR1YyWlc1MExtZGxkRUZ3Y0Y5cFpDZ3BPdzBLRFFvZ0lHbG1JQ2h3YkdGMFptOXliU0E5UFNBaWMyVnlkbVZ5SWlBbUppQmhjSEJKWkNBaFBTQWljMlZqY21WMElpa2dldzBLSUNBZ0lIUm9jbTkzSUNKVFpYSjJaWEl0YzJsa1pTQmxkbVZ1ZENCb1lYTWdhVzUyWVd4cFpDQmhjSEJmYVdRNklDSWdLeUJoY0hCSlpEc05DaUFnZlEwS0lDQU5DaUFnYVdZZ0tHRndjRWxrSUQwOUlHNTFiR3dwSUhzTkNpQWdJQ0J5WlhSMWNtNGdXMTA3RFFvZ0lIME5DZzBLSUNBdkx5QlZjMlVnYm1WM0lGTjBjbWx1WnlncElHSmxZMkYxYzJVZ2FIUjBjRG92TDI1bGJITnZibmRsYkd4ekxtNWxkQzh5TURFeUx6QXlMMnB6YjI0dGMzUnlhVzVuYVdaNUxYZHBkR2d0YldGd2NHVmtMWFpoY21saFlteGxjeThOQ2lBZ2RtRnlJR0Z3Y0Vsa1ZYQndaWElnUFNCdVpYY2dVM1J5YVc1bktHRndjRWxrTG5SdlZYQndaWEpEWVhObEtDa3BPdzBLRFFvZ0lISmxkSFZ5YmlCYklIc2djMk5vWlcxaE9pQWlhV2RzZFRwamIyMHVZV050WlM5bWIyOHZhbk52Ym5OamFHVnRZUzh4TFRBdE1DSXNEUW9nSUNBZ0lDQWdJQ0FnSUNBZ0lDQmtZWFJoT2lCN0lHRndjRWxrVlhCd1pYSTZJR0Z3Y0Vsa1ZYQndaWElnZlEwS0lDQWdJQ0FnSUNBZ0lDQjlJRjA3RFFwOSJ9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91c2VyX2FnZW50X3V0aWxzX2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3ciLCJuYW1lIjoidXNlcl9hZ2VudF91dGlsc19jb25maWciLCJlbmFibGVkIjp0cnVlLCJwYXJhbWV0ZXJzIjp7fX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvY2FtcGFpZ25fYXR0cmlidXRpb24vanNvbnNjaGVtYS8xLTAtMSIsImRhdGEiOnsibmFtZSI6ImNhbXBhaWduX2F0dHJpYnV0aW9uIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJtYXBwaW5nIjoic3RhdGljIiwiZmllbGRzIjp7Im1rdE1lZGl1bSI6WyJ1dG1fbWVkaXVtIl0sIm1rdFNvdXJjZSI6WyJ1dG1fc291cmNlIl0sIm1rdFRlcm0iOlsidXRtX3Rlcm0iXSwibWt0Q29udGVudCI6WyJ1dG1fY29udGVudCJdLCJta3RDYW1wYWlnbiI6WyJ1dG1fY2FtcGFpZ24iXX19fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cy9hcGlfcmVxdWVzdF9lbnJpY2htZW50X2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cuZW5yaWNobWVudHMiLCJuYW1lIjoiYXBpX3JlcXVlc3RfZW5yaWNobWVudF9jb25maWciLCJlbmFibGVkIjpmYWxzZSwicGFyYW1ldGVycyI6eyJpbnB1dHMiOlt7ImtleSI6InVzZXIiLCJwb2pvIjp7ImZpZWxkIjoidXNlcl9pZCJ9fSx7ImtleSI6InVzZXIiLCJqc29uIjp7ImZpZWxkIjoiY29udGV4dHMiLCJzY2hlbWFDcml0ZXJpb24iOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jbGllbnRfc2Vzc2lvbi9qc29uc2NoZW1hLzEtKi0qIiwianNvblBhdGgiOiIkLnVzZXJJZCJ9fSx7ImtleSI6ImNsaWVudCIsInBvam8iOnsiZmllbGQiOiJhcHBfaWQifX1dLCJhcGkiOnsiaHR0cCI6eyJtZXRob2QiOiJHRVQiLCJ1cmkiOiJodHRwOi8vYXBpLmFjbWUuY29tL3VzZXJzL3t7Y2xpZW50fX0ve3t1c2VyfX0/Zm9ybWF0PWpzb24iLCJ0aW1lb3V0IjoyMDAwLCJhdXRoZW50aWNhdGlvbiI6eyJodHRwQmFzaWMiOnsidXNlcm5hbWUiOiJ4eHgiLCJwYXNzd29yZCI6Inl5eSJ9fX19LCJvdXRwdXRzIjpbeyJzY2hlbWEiOiJpZ2x1OmNvbS5hY21lL3VzZXIvanNvbnNjaGVtYS8xLTAtMCIsImpzb24iOnsianNvblBhdGgiOiIkLnJlY29yZCJ9fV0sImNhY2hlIjp7InNpemUiOjMwMDAsInR0bCI6NjB9fX19XX0= --input-folder hdfs:///local/snowplow/raw-events/* --output-folder hdfs:///local/snowplow/enriched-events/ --bad-folder s3://infinigrow-logs/etlrunner/enriched/bad/run=2018-10-04-04-30-22/
> 2018-10-04T04:36:30.087Z INFO HadoopJarStepRunner.Runner: startRun() called for s-14Q6W9V3K08FF Child Pid: 8876
> INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
> INFO Process still running
> INFO Process still running
> INFO Process still running
> INFO Process still running
> INFO Process still running
> ...
> ...
> ...
strerr:
Warning: Skip remote jar s3://snowplow-hosted-assets-us-west-2/3-enrich/spark-enrich/snowplow-spark-enrich-1.14.0.jar.
18/10/04 04:36:34 INFO RMProxy: Connecting to ResourceManager at ip-172-31-18-60.us-west-2.compute.internal/172.31.18.60:8032
18/10/04 04:36:34 INFO Client: Requesting a new application from cluster with 2 NodeManagers
18/10/04 04:36:34 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (6144 MB per container)
18/10/04 04:36:34 INFO Client: Will allocate AM container, with 6143 MB memory including 558 MB overhead
18/10/04 04:36:34 INFO Client: Setting up container launch context for our AM
18/10/04 04:36:34 INFO Client: Setting up the launch environment for our AM container
18/10/04 04:36:34 INFO Client: Preparing resources for our AM container
18/10/04 04:36:36 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
18/10/04 04:36:38 INFO Client: Uploading resource file:/mnt/tmp/spark-778585da-c362-4270-bab7-4598895a7872/__spark_libs__7664507486617369497.zip -> hdfs://ip-172-31-18-60.us-west-2.compute.internal:8020/user/hadoop/.sparkStaging/application_1538627609524_0003/__spark_libs__7664507486617369497.zip
18/10/04 04:36:41 WARN RoleMappings: Found no mappings configured with 'fs.s3.authorization.roleMapping', credentials resolution may not work as expected
18/10/04 04:36:41 INFO Client: Uploading resource s3://snowplow-hosted-assets-us-west-2/3-enrich/spark-enrich/snowplow-spark-enrich-1.14.0.jar -> hdfs://ip-172-31-18-60.us-west-2.compute.internal:8020/user/hadoop/.sparkStaging/application_1538627609524_0003/snowplow-spark-enrich-1.14.0.jar
18/10/04 04:36:41 INFO S3NativeFileSystem: Opening 's3://snowplow-hosted-assets-us-west-2/3-enrich/spark-enrich/snowplow-spark-enrich-1.14.0.jar' for reading
18/10/04 04:36:43 INFO Client: Uploading resource file:/mnt/tmp/spark-778585da-c362-4270-bab7-4598895a7872/__spark_conf__1278190290174767064.zip -> hdfs://ip-172-31-18-60.us-west-2.compute.internal:8020/user/hadoop/.sparkStaging/application_1538627609524_0003/__spark_conf__.zip
18/10/04 04:36:43 INFO SecurityManager: Changing view acls to: hadoop
18/10/04 04:36:43 INFO SecurityManager: Changing modify acls to: hadoop
18/10/04 04:36:43 INFO SecurityManager: Changing view acls groups to:
18/10/04 04:36:43 INFO SecurityManager: Changing modify acls groups to:
18/10/04 04:36:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
18/10/04 04:36:43 INFO Client: Submitting application application_1538627609524_0003 to ResourceManager
18/10/04 04:36:43 INFO YarnClientImpl: Submitted application application_1538627609524_0003
18/10/04 04:36:44 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
18/10/04 04:36:44 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1538627803560
final status: UNDEFINED
tracking URL: http://ip-172-31-18-60.us-west-2.compute.internal:20888/proxy/application_1538627609524_0003/
user: hadoop
18/10/04 04:36:45 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
18/10/04 04:36:46 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
18/10/04 04:36:47 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
18/10/04 04:36:48 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
18/10/04 04:36:49 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
18/10/04 04:36:50 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
18/10/04 04:36:51 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
18/10/04 04:36:52 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
18/10/04 04:36:52 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 172.31.28.194
ApplicationMaster RPC port: 0
queue: default
start time: 1538627803560
final status: UNDEFINED
tracking URL: http://ip-172-31-18-60.us-west-2.compute.internal:20888/proxy/application_1538627609524_0003/
user: hadoop
18/10/04 04:36:53 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
18/10/04 04:36:54 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
18/10/04 04:36:55 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
18/10/04 04:36:56 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
18/10/04 04:36:57 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
18/10/04 04:36:58 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
18/10/04 04:36:59 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
18/10/04 04:37:00 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
18/10/04 04:37:01 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
log from spark console:
org.apache.spark.sql.DataFrameWriter.text(DataFrameWriter.scala:555)
com.snowplowanalytics.snowplow.enrich.spark.EnrichJob.run(EnrichJob.scala:212)
com.snowplowanalytics.snowplow.enrich.spark.EnrichJob$.run(EnrichJob.scala:88)
com.snowplowanalytics.snowplow.enrich.spark.SparkJob$class.main(SparkJob.scala:33)
com.snowplowanalytics.snowplow.enrich.spark.EnrichJob$.main(EnrichJob.scala:52)
com.snowplowanalytics.snowplow.enrich.spark.EnrichJob.main(EnrichJob.scala)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:635)
What can cause this? Also is it risky to cancel the cluster and run it again? Might cause losing data?