Archive shred step is not working

Our EMR jobs are not starting since 2-3 days. When I manually started a job it is showing there are contents in shredded good so I ran job using –resume-from archive_shredded but it is actually not working, though EMR job shows completed there are still files in shredded good bucket

@Dev what version are you on? i had the same problem. upgrading will fix it. i’m assuming it finished all steps previously and when you try to run a new full run it fails because of that error above. the problem could be because there are too many $folder in your directory.

this script helps too.

1 Like