I think this may occur when there’s an issue accessing the S3 bucket in question. In your config.yml ensure that all string values are enclosed in quotes and try rerunning the emr-etl-runner.
Hi Mike,
bellow is my configuration, all string values are enclosed in quotes but still getting error.
aws:
access_key_id: key
emr:
additional_info: ~
ami_version: “4.5.0”
bootstrap:
bootstrap_failure_tries: 3
ec2_key_name: snowplow
ec2_subnet_id: subnet-0cxxxxxx
jobflow:
core_instance_count: 2
core_instance_ebs:
ebs_optimized: false
volume_iops: 400
volume_size: 100
volume_type: gp2
core_instance_type: m1.medium
master_instance_type: m1.medium
task_instance_bid: 0.015
task_instance_count: 0
task_instance_type: m1.medium
jobflow_role: EMR_EC2_DefaultRole
placement: ~
region: us-west-2
service_role: EMR_DefaultRole
software:
hbase: ~
lingual: ~
s3:
buckets:
assets: “s3://snowplow-hosted-assets”
enriched:
archive: “s3://snowplow-logs-xxx/enriched”
bad: “s3://snowplow-logs-xxx/enrich/bad”
errors: “s3://snowplow-logs-xxx/enrich/errors”
good: “s3://snowplow-logs-xxx/enrich/good”
jsonpath_assets: ~
log: “s3://snowplow-logs-xxx/slog/”
raw:
archive: “s3://snowplow-logs-xxx/archive/raw”
in:
- “s3://snowplow-logs-xxx/logs”
processing: “s3://snowplow-logs-xxx/raw/pro1”
shredded:
archive: “s3://snowplow-logs-xxx/archive/shredded”
bad: “s3://snowplow-logs-xxx/shredded/bad”
errors: “s3://snowplow-logs-xxx/shredded/errors”
good: “s3://snowplow-logs-xxx/shredded/good”
region: us-west-2
secret_access_key: key
collectors:
format: thrift
enrich:
continue_on_unexpected_error: false
job_name: “Snowplow ETL”
output_compression: NONE
versions:
hadoop_elasticsearch: “0.1.0”
hadoop_enrich: “1.8.0”
hadoop_shred: “0.10.0”
monitoring:
logging:
level: DEBUG
snowplow:
app_id: emrapp
collector: sp.cannymedianetwork.com
method: get
tags: {}
storage:
download:
folder: ~
targets:
-
comprows: 200000
database: snowplow
host: sp.xxxxxxxxxxx.us-west-2.redshift.amazonaws.com
maxerror: 1
name: sp
password: password
port: 5439
ssl_mode: disable
table: atomic.events
type: redshift
username: username
Are your quotes being escaped when posting here? For example the region us-west-2 should appear in quotes as "us-west-2".