R35 RDB Shredder config.hocon env var error with dataflow-runner

For engineers
1

Hi,
I’m running the rdb shredder using dataflow-runner and gets the following error:

Unable to parse the configuration: Could not resolve substitution to a value: ${S3_SHREDDED_OUTPUT_GOOD}.

this line is from the hocon.config file and the wanted environment variable is set on my docker container running the dataflow-runner.

if I understand it correctly, I need the emr to have all the used environments from the hocon file.
Is there a way doing so? or a better way of good practice to make it work?

  • the config.hocon and the wanted env vars are taken from a cm, which the rdb loader creates

Edit: I have tryed to add the environment variables to the emr but it still fails with the same error.
is there any best practice on how to do so? or just rewrite/inject the config.hocon with the hardcoded values

2

were you able to solve this? I also hit similar problem