With the data (LZO) living in that bucket, running my EMR runner:
E, [2018-09-26T21:14:42.121423 #4374] ERROR – : No run folders in [s3://piv-stream-data-prod-bucket/] found
I am running:
[ec2-user@ip-172-31-8-153 ~]$ ./snowplow-emr-etl-runner run -c emr.yaml -r resolver.json -t targets -d -f shred
targets has correct redshift config, and relevant emr:
enriched:
good: s3://piv-stream-data-prod-bucket/ # e.g. s3://my-out-bucket/enriched/good
bad: s3://piv-stream-data-prod-bucket-bad/ # e.g. s3://my-out-bucket/enriched/bad
errors: # Leave blank unless :continue_on_unexpected_error: set to true below
archive: # Where to archive enriched events to, e.g. s3://my-archive-bucket/enriched
stream: s3://piv-stream-data-prod-bucket # stream bucket
shredded:
good: s3://piv-prod-shredded-good # e.g. s3://my-out-bucket/shredded/good
bad: s3://piv-prod-shredded-bad # e.g. s3://my-out-bucket/shredded/bad
errors: # Leave blank unless :continue_on_unexpected_error: set to true below
archive: # Where to archive shredded events to, e.g. s3://my-archive-bucket/shredded
There is clearly compressed s3 loaded data in that bucket, what am I doing wrong?