Batch pipeline upgrade from R106 to R109 failed

For engineers AWS batch pipeline (Legacy)
1

Hi
I’m testing the upgrade from R106 to R109.
I’m trying to process old events with R109 & loading them to a temp redshift and then compare it with the events processed with R106.

attached the config files I’m using with R109 (I did minimal changes, did not use s3 enryption but i did change the user agent enrichment to the default sample in GIT, I also did npot upgrade the RDB loader).

the EMR job don’t even starts, it immediately fails with “AWS EMR API Error (ValidationException): Size of step parameter length exceeded the maximum allowed”.

I’m executing it using snowplow-emr-etl-runner run --config config.yml --enrichments enrichments --resolver iglu_resolver.json --targets targets --lock /temp/local_lock

config
config.png713×781 26.4 KB
resolver
any idea what is the issue?

Thanks a lot in advance,
Moran

2

@morans, the error comes from exceeding the total length of the string submitted as a command to EMR. You could try to minify your enrichments files content and/or resolver configuration (basically any configuration JSONs) and thus reduce the total length of the string submitted.