When snowplow-emr-etlrunner finds no new files to process in aws.s3.buckets.raw.in, it returns exit code 3 and logs to debug: “No logs to process: No Snowplow logs to process since last run”.
I am running snowplow-emr-etlrunner as a step in a bash script in a cron job. The script starts with “set -e” so that if any step fails (exit code != 0), subsequent steps will be canceled. In the case of no new data, I would like for the wrapper script to continue and run the other steps.
I can wrap the call to snowplow-emr-etlrunner in a conditional so it will ignore exit code 3 and keep running if there is no new data; but before I do that, thought I would check here to see if
-
Is there a way to configure snowplow-emr-etlrunner so it will return 0 on empty input buckets?
-
Are there other conditions that lead to exit code 3 that are less innocuous?