Hello,
I had a few problem in EMRETLRunner. Here is that link to the problem. I got the solution for that problem. This job got failed in the processing stage. If i re ran this job, will all the logs will be moved to the redshift. I have few more raw logs. Will that raw logs processed during this re-run. I want all the logs to be moved to the redshift. In what direction i have to look into it. What are the step i have to follow to achieve this ?.
All pointers to solution much appreciated.
Hi @rajan,
Please, review the diagram and instructions on Batch Pipeline Steps wiki page.
In short, if you have any files in your processing bucket (due to failure at that stage), you would need to rerun the job with --skip staging option.
Once those files have been processed successfully (including loading to Redshift), you can run the job again as usual (without --skip option) to pick up the remaining “raw” files.
The point is the job will not start from the very beginning if there are files in any of the following buckets:
processing (cleared by EMR job)enriched/good (cleared by StorageLoader)shredded/good (cleared by StorageLoader)They are all cleared during EMR and StorageLoader job runs. Any remaining files would indicate a job failure at some stage.
Again, please, follow the instructions on the wiki page mentioned above. It shows you how to rerun the job depending on the failed step.
–Ihor
Thank you for support. I was finally able to get ETLEMR running successfully.