Hi,
we updated from snowplow/rdb-loader-redshift:5.4.0 to 5.4.1 yesterday and now we get a strange
Unexpected argument /bin/sh error on Cloudwatch.
Did something change in the folder structure?
.-M
Hi @mgloel, nothing changed in the folder structure between 5.4.0 and 5.4.1.
From the logs you shared, it looks like you have passed “/bin/sh” and the CMD argument to the docker container. This won’t work, because the container entrypoint is a script that runs the loader application, and the CMD should be used just for command line parameters expected by the loader.
Could you share details of how you are running the docker container? I might be able to spot what is wrong.
FROM snowplow/rdb-loader-redshift:5.4.3
COPY ./src/config.hocon /snowplow/config.hocon
COPY ./src/resolver.json /snowplow/resolver.json
ARG STAGE
ARG AWS_ACCOUNT_NUMBER
ARG AWS_DEFAULT_REGION
ARG SP_SCHEMA_URI
ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
ARG DB_REDSHIFT_PASSWORD
ARG SQS_QUEUE
ARG REDSHIFTLOADROLE
ARG REDSHIFT_ENDPOINT_ID
ARG REDSHIFT_ENDPOINT="sp-redshift-cluster-$STAGE.$REDSHIFT_ENDPOINT_ID.$AWS_DEFAULT_REGION.redshift.amazonaws.com"
ENV REDSHIFT_ENDPOINT=$REDSHIFT_ENDPOINT
ENV REDSHIFT_ENDPOINT_ID=$REDSHIFT_ENDPOINT_ID
ENV AWS_REGION=$AWS_DEFAULT_REGION
ENV AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
ENV AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
ENV SQS_QUEUE=$SQS_QUEUE
ENV STAGE=$STAGE
ENV SP_SCHEMA_URI=$SP_SCHEMA_URI
ENV DB_REDSHIFT_PASSWORD=$DB_REDSHIFT_PASSWORD
ENV REDSHIFTLOADROLE=$REDSHIFTLOADROLE
COPY src/ .
USER root
RUN sh modify_configs.sh
CMD /opt/snowplow/bin/snowplow-redshift-loader \
--iglu-config $(cat /snowplow/resolver.json | base64 -w 0) \
--config $(cat /snowplow/config.hocon | base64 -w 0)
Where the shell script basically substitutes the strings representing environment variables with their values (with sed command)
The problem persists with 5.4.3
1 Like
Fixed the issue.
I changed the entrypoint to in the Dockerfile with ENTRYPOINT ["/usr/bin/env"]. This overrode the default entrypoint /bin/sh -c that is used in docker unless specified otherwise.
Setting the entrypoint now to /usr/bin/env looks up the PATH environment variable and take the executable at the right location . So for example, if we need to run sh , we don’t need to hardcode the absolute path to it and the problem arises if it lives elsewhere. The PATH variable takes care of finding sh at its location and call it from there.
Even though it fixed the issue for us, I still don’t understand why we got this error message in the first place. I understand that /opt/snowplow/bin/snowplow-redshift-loader is a bash script with the shebang #!/usr/bin/env bash at the top so it should invoke bash at its correct location to execute the script. Why it complained about /bin/sh still remains unclear to me.
2 Likes
Hi @Kristina_Pianykh it’s great that you fixed the issue, and thank you for sharing that you found the solution.
It was interesting to me that you had customised the snowplow docker image with your own dockerfile. Of course you are free to do this, but it is our intention that you should be able to run the standard docker image, without modification. Our intention is that configuration parameters should be provided at run time, rather than baked into the image.
May I ask why you chose to modify the standard docker image? Was it maybe because RDB loader did not have a user-friendly way of providing configuration via environment variables? If that was your reason then it’s a very good reason! I would probably have done the same thing.
We recently released RDB loader version 5.5.0 and it has some improvements to how it receives configuration. You no longer need to provide base64-encoded strings on the command line. And it is possible to provide a hocon config file in which you template environment variables, e.g.: password: ${DB_REDSHIFT_PASSWORD}
I’m hoping that with the new version, we can find a way for you to run the standard image without any customisation.
3 Likes
Hi @istreeter ,
yes, that was our reason for modifying it.
Thanks for the update, we will give version 5.5.0 a shot. Sounds amazing.
Best,
M.