Hi @pramod.niralakeri , I agree this should work if the SHRED_BUCKET environment variable really is set properly. So my first guess is that something has gone wrong setting the environment variable.
Are you running the shredder on EMR? If yes, then it is the EMR cluster that needs to have the environment variable set, not dataflow runner. Please can you share how you are setting the environment variable?
Hi @pramod.niralakeri I’m afraid I don’t know how to set an environment variable in EMR. I had a quick search for an answer, but I couldn’t find an answer.
The expected way to deploy the shredder is not to rely on environment variables in the hocon. But instead, you can template the variables into your config file before running dataflow-runner. Then, dataflow runner can read the templated configuration file (including the bucket name) and submit the steps to EMR using valid shredder base64-encoded config.