I’m running RDB shredder and it’s failing at step 2.
followed this doc to bring up the service. My config looks like below.
cluster.json
{
"schema": "iglu:com.snowplowanalytics.dataflowrunner/ClusterConfig/avro/1-1-0",
"data": {
"name": "RDB Shredder",
"logUri": "s3://snowplow-s3-emr-logs/",
"region": "ap-southeast-1",
"credentials": {
"accessKeyId": "123",
"secretAccessKey": "123"
},
"roles": {
"jobflow": "EMR_EC2_DefaultRole",
"service": "EMR_DefaultRole"
},
"ec2": {
"amiVersion": "6.2.0",
"keyName": "xyz-ap",
"location": {
"vpc": {
"subnetId": "subnet-123"
}
},
"instances": {
"master": {
"type": "m4.large",
"ebsConfiguration": {
"ebsOptimized": true,
"ebsBlockDeviceConfigs": [
]
}
},
"core": {
"type": "r4.xlarge",
"count": 1
},
"task": {
"type": "m4.large",
"count": 1,
"bid": "0.015"
}
}
},
"tags": [],
"bootstrapActionConfigs": [],
"configurations": [{
"classification": "core-site",
"properties": {
"Io.file.buffer.size": "65536"
},
"configurations": [
]
},
{
"classification": "yarn-site",
"properties": {
"yarn.nodemanager.resource.memory-mb": "57344",
"yarn.scheduler.maximum-allocation-mb": "57344",
"yarn.nodemanager.vmem-check-enabled": "false"
},
"configurations": [
]
},
{
"classification": "spark",
"properties": {
"maximizeResourceAllocation": "false"
},
"configurations": [
]
},
{
"classification": "spark-defaults",
"properties": {
"spark.executor.memory": "7G",
"spark.driver.memory": "7G",
"spark.driver.cores": "3",
"spark.default.parallelism": "24",
"spark.executor.cores": "1",
"spark.executor.instances": "6",
"spark.dynamicAllocation.enabled": "false"
},
"configurations": [
]
}
],
"applications": ["Hadoop", "Spark"]
}
}
paybook.json
{
"schema": "iglu:com.snowplowanalytics.dataflowrunner/PlaybookConfig/avro/1-0-1",
"data": {
"region": "ap-southeast-1",
"credentials": {
"accessKeyId": "123",
"secretAccessKey": "123"
},
"steps": [{
"type": "CUSTOM_JAR",
"name": "S3DistCp enriched data archiving",
"actionOnFailure": "CANCEL_AND_WAIT",
"jar": "/usr/share/aws/emr/s3-dist-cp/lib/s3-dist-cp.jar",
"arguments": [
"--src", "s3://snowplow-s3-sink/enriched/",
"--dest", "s3://snowplow-s3-shred-test/shredded/archive/run=2006-01-02-15-04-05",
"--s3Endpoint", "s3-ap-southeast-1.amazonaws.com",
"--srcPattern", ".*",
"--outputCodec", "gz",
"--deleteOnSuccess"
]
},
{
"type": "CUSTOM_JAR",
"name": "RDB Shredder",
"actionOnFailure": "CANCEL_AND_WAIT",
"jar": "command-runner.jar",
"arguments": [
"spark-submit",
"--class",
"com.snowplowanalytics.snowplow.rdbloader.shredder.batch.Main",
"--master", "yarn",
"--deploy-mode", "cluster",
"s3://snowplow-hosted-assets-eu-central-1/4-storage/rdb-shredder/snowplow-rdb-shredder-1.1.0.jar",
"--iglu-config", "Z2x1OmNvbS5zbm93cGxvd2FuYWVyLWNvbmZpZewogICAgInNjaGVtYSI6ICJpy9qc29uc2NoZW1hLzEtMC0xIiwKICAgICJkYXRhIjogewogICAgICAiY2FjaGVTaXplIjogNTAwLAogICAgICAicmVwb3NpdG9yaWVzIjogWwogICAgICAgIHsKICAgICAgICAgICJuYW1lIjogIklnbHUgQ2VudHJhbCIsCiAgICAgICAgICAicHJpb3JpdHkiOiAwLAogICAgICAgICAgInZlbmRvclByZWZpeGVzIjogWyAiY29tLnNub3dwbG93YW5hbHl0aWNzIiBdLAogICAgICAgICAgImNvbm5lY3Rpb24iOiB7CiAgICAgICAgICAgICJodHRwIjogewogICAgICAgICAgICAgICJ1cmkiOiAiaHR0cDovL2lnbHVjZW50cmFsLmNvbSIKICAgICAgICAgICAgfQogICAgICAgICAgfQogICAgICAgIH0KICAgICAgXQogICAgfQp9",
"--config", "ewogICMgUGF0aCB0byBlGlyZWN0b3JpZXMpIGZvciBTMyBpbnB1dAogICJpbnB1dCI6ICJzMzoewogICAgInNjaGVtYSI6ICJpvL251Y2xldewogICAgInNjaGVtYSI6ICJpXMtZGF0YS10ZWFtLXNub3dwbG93LWthZmthLewogICAgInNjaGVtYSI6ICJpXMzLXNpbmstdGVzdC9lbnJpY2hlZC8iLAoKICAjIFBhdGggdG8gc2hyZWRkZWQgYXJjaGl2ZQogICJvdXRwdXQiOiB7CiAgICAjIFBhdGggdG8gc2hyZWRkZWQgb3V0cHV0CiAgICAicGF0aCI6ICJzMzovL251Y2xldXMtZGF0YS10ZWFtLXNub3dwbG93LWthZmthLXMzLXNocmVkLXRlc3Qvc2hyZWRkZWQvIiwKICAgICMgU2hyZWRkZXIgb3V0cHV0IGNvbXByZXNzaW9uLCBHWklQIG9yIE5PTkUKICAgICMgT3B0aW9uYWwsIGRlZmF1bHQgdmFsdWUgR1pJUAogICAgImNvbXByZXNzaW9uIjogIkdaSVAiLAogICAgIyBUaGlzIGZpZWxkIGlzIG9wdGlvbmFsIGlmIGl0IGNhbiBiZSByZXNvbHZlZCB3aXRoIEFXUyByZWdpb24gcHJvdmlkZXIgY2hhaW4uCiAgICAjIEl0IGNoZWNrcyBwbGFjZXMgbGlrZSBlbnYgdmFyaWFibGVzLCBzeXN0ZW0gcHJvcGVydGllcywgQVdTIHByb2ZpbGUgZmlsZS4KICAgICMgaHR0cHM6Ly9zZGsuYW1hem9uYXdzLmNvbS9qYXZhL2FwaS9sYXRlc3Qvc29mdHdhcmUvYW1hem9uL2F3c3Nkay9yZWdpb25zL3Byb3ZpZGVycy9EZWZhdWx0QXdzUmVnaW9uUHJvdmlkZXJDaGFpbi5odG1sCiAgICAicmVnaW9uIjogImFwLXNvdXRoZWFzdC0xIgogIH0KCiAgIyBRdWV1ZSB1c2VkIHRvIGNvbW11bmljYXRlIHdpdGggTG9hZGVyCiAgInF1ZXVlIjogewogICAgIyBUeXBlIG9mIHRoZSBxdWV1ZS4gSXQgY2FuIGJlIGVpdGhlciBzcXMgb3Igc25zCiAgICAidHlwZSI6ICJzcXMiLAogICAgIyBOYW1lIG9mIHRoZSBzcXMgcXVldWUKICAgICJxdWV1ZU5hbWUiOiAic25vd3Bsb3ctYmlrcm95LXNocmVkZGVyLWxvYWRlciIsCiAgICAjIFJlZ2lvbiBvZiB0aGUgU1FTIHF1ZXVlLgogICAgIyBPcHRpb25hbCBpZiBpdCBjYW4gYmUgcmVzb2x2ZWQgd2l0aCBBV1MgcmVnaW9uIHByb3ZpZGVyIGNoYWluLgogICAgInJlZ2lvbiI6ICJhcC1zb3V0aGVhc3QtMSIKICB9CiAgIyBTTlMgZXhhbXBsZToKICAjInF1ZXVlIjogewogICMgICMgVHlwZSBvZiB0aGUgcXVldWUuIEl0IGNhbiBiZSBlaXRoZXIgc3FzIG9yIHNucwogICMgICJ0eXBlIjogInNucyIsCiAgIyAgIyBBUk4gb2YgU05TIHRvcGljCiAgIyAgInRvcGljQXJuIjogImFybjphd3M6c25zOmV1LWNlbnRyYWwtMToxMjM0NTY3ODk6dGVzdC1zbnMtdG9waWMiLAogICMgICMgUmVnaW9uIG9mIHRoZSBTTlMgdG9waWMKICAjICAicmVnaW9uIjogImV1LWNlbnRyYWwtMSIKICAjfQoKICAjIFNjaGVtYS1zcGVjaWZpYyBmb3JtYXQgc2V0dGluZ3MgKHJlY29tbWVuZGVkIHRvIGxlYXZlIGFsbCB0aHJlZSBncm91cHMgZW1wdHkgYW5kIHVzZSBUU1YgYXMgZGVmYXVsdCkKICAiZm9ybWF0cyI6IHsKICAgICMgRm9ybWF0IHVzZWQgYnkgZGVmYXVsdCAoVFNWIG9yIEpTT04pCiAgICAjIE9wdGlvbmFsLCBkZWZhdWx0IHZhbHVlIFRTVgogICAgImRlZmF1bHQiOiAiVFNWIiwKICAgICMgU2NoZW1hcyB0byBiZSBzaHJlZGRlZCBhcyBKU09OcywgY29ycmVzcG9uZGluZyBKU09OUGF0aCBmaWxlcyBtdXN0IGJlIHByZXNlbnQuIEF1dG9taWdyYXRpb25zIHdpbGwgYmUgZGlzYWJsZWQKICAgICMgT3B0aW9uYWwsIGRlZmF1bHQgdmFsdWUgW10KICAgICJqc29uIjogWwogICAgICAiaWdsdTpjb20uYWNtZS9qc29uLWV2ZW50L2pzb25zY2hlbWEvMS0wLTAiLAogICAgICAiaWdsdTpjb20uYWNtZS9qc29uLWV2ZW50L2pzb25zY2hlbWEvMi0qLSoiCiAgICBdLAogICAgIyBTY2hlbWFzIHRvIGJlIHNocmVkZGVkIGFzIFRTVnMsIHByZXNlbmNlIG9mIHRoZSBzY2hlbWEgb24gSWdsdSBTZXJ2ZXIgaXMgbmVjZXNzYXJ5LiBBdXRvbWlnYXJ0aW9ucyBlbmFibGVkCiAgICAjIE9wdGlvbmFsLCBkZWZhdWx0IHZhbHVlIFtdCiAgICAidHN2IjogWyBdLAogICAgIyBTY2hlbWFzIHRoYXQgd29uJ3QgYmUgbG9hZGVkCiAgICAjIE9wdGlvbmFsLCBkZWZhdWx0IHZhbHVlIFtdCiAgICAic2tpcCI6IFsKICAgICAgImlnbHU6Y29tLmFjbWUvc2tpcC1ldmVudC9qc29uc2NoZW1hLzEtKi0qIgogICAgXQogIH0sCgogICMgT2JzZXJ2YWJpbGl0eSBhbmQgcmVwb3J0aW5nIG9wdGlvbnMKICAibW9uaXRvcmluZyI6IHsKICAgICMgT3B0aW9uYWwsIGZvciB0cmFja2luZyBydW50aW1lIGV4Y2VwdGlvbnMKICAgICJzZW50cnkiOiB7CiAgICAgICJkc24iOiAiaHR0cDovL3NlbnRyeS5hY21lLmNvbSIKICAgIH0KICB9Cn0K"
]
}
],
"tags": []
}
}
command to run
dataflow-runner run-transient --emr-config ./config/cluster.json --emr-playbook ./config/playbook.json
Interesting logs (where I feel the issue is)
22/01/01 08:31:47 INFO ApplicationMaster: Waiting for spark context initialization...
- Unable to parse the configuration: DecodingFailure at .name: Attempt to decode value on failed cursor.
Usage: snowplow-rdb-shredder-1.1.0 --iglu-config <<base64>> [--duplicate-storage-config <<base64>>] --config <config.hocon>
Spark job to shred event and context JSONs from Snowplow enriched events
Options and flags:
--help
Display this help text.
--iglu-config <<base64>>
Base64-encoded Iglu Client JSON config
--duplicate-storage-config <<base64>>
Base64-encoded Events Manifest JSON config
--config <config.hocon>, -c <config.hocon>
base64-encoded config HOCON
22/01/01 08:31:51 INFO ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: Shutdown hook called before final status was reported.)
22/01/01 08:31:51 INFO ApplicationMaster: Deleting staging directory hdfs://ip-x-x-x-x.ap-southeast-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1641025694876_0002
22/01/01 08:31:52 INFO ShutdownHookManager: Shutdown hook called
For more detailed output, check the application tracking page: http://ip-x-x-x-x.ap-southeast-1.compute.internal:8088/cluster/app/application_1641025694876_0002 Then click on links to logs of each attempt.
. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1641025896332
final status: FAILED
tracking URL: http://ip-x-x-x-x.ap-southeast-1.compute.internal:8088/cluster/app/application_1641025694876_0002
user: hadoop
22/01/01 08:31:53 ERROR Client: Application diagnostics message: Application application_1641025694876_0002 failed 2 times due to AM Container for appattempt_1641025694876_0002_000002 exited with exitCode: 2
Failing this attempt.Diagnostics: [2022-01-01 08:31:52.508]Exception from container-launch.
Container id: container_1641025694876_0002_02_000001
Exit code: 2
[2022-01-01 08:31:52.511]Container exited with a non-zero exit code 2. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Now observe this error
- Unable to parse the configuration: DecodingFailure at .name: Attempt to decode value on failed cursor
from above error logs. Not sure whats wrong in .name (playbook.json) config or somewhere around it in 2nd step.
Versions.
DATAFLOW_RUNNER_VERSION=0.6.0-rc2
snowplow-s3-loader:2.1.3
stream-enrich-kinesis:2.0.5
scala-stream-collector-kinesis:2.4.5
help is appreciated. thanks
