BigQuery StreamLoader Custom Docker Image

For engineers GCP pipeline
1

Hey,

I’m trying to run the bigquery streamloader on GCP’s appengine service for which im trying to create a custom docker image.

When i run the jar file on my local system the bigquery streamloader runs fine, command for the same:
java -jar snowplow-bigquery-streamloader-1.0.1.jar --config $(cat config.hocon | base64 -w 0) --resolver $(cat resolver.json | base64 -w 0)

But when i run the same command as docker, i get the following issue:
image

Can someone help me out?

Dockerfile is as follows:

FROM openjdk:18-jdk-alpine

COPY snowplow-bigquery-streamloader-1.0.1.jar snowplow-bigquery-streamloader-1.0.1.jar
COPY config.hocon config.hocon
COPY resolver.json resolver.json

CMD ["java","-jar","snowplow-bigquery-streamloader-1.0.1.jar","--config","$(cat config.hocon | base64 -w 0)","--resolver","$(cat resolver.json | base64 -w 0)"]

Thank you!

2

Hi @siv, I think what’s going on here is that docker’s CMD does not support command substitutions like $(cat config.hocon | base64 -w 0) in the dockerfile. To get this working, you would have to literally copy the entire base64-encoded config into your CMD:

FROM openjdk:18-jdk-alpine

COPY snowplow-bigquery-streamloader-1.0.1.jar snowplow-bigquery-streamloader-1.0.1.jar

CMD [ \
  "java", \
  "-jar", \
  "snowplow-bigquery-streamloader-1.0.1.jar", \
  "--config", \
  "ewogICJwcm9qZWN0SWQiOiAiY29tLWFjbWUiCgogICJsb2FkZXIiOiB7CiAgICAiaW5wdXQiOiB7CiAgICAgICJzdWJzY3JpcHRpb24iOiAiZW5yaWNoZWQtc3ViIgogICAgfQoKICAgICJvdXRwdXQiOiB7CiAg...... etc", \
  "--resolver", \
  "ewogICJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3MuaWdsdS9yZXNvbHZlci1jb25maWcvanNvbnNj... etc" \
]

An easier alternative method might be to add a bash script to your docker image, which runs the java command.

COPY run-loader.sh run-loader-sh
CMD ["run-loader.sh"]

The recommended way to run the bigquery loader is using the provided docker image and providing configuration at run time, rather than building a custom image and providing configuration at build time. People using the recommended method won’t have the problem you’re facing here.

2 Likes
3

Hi @istreeter

Thank you for your response, i did use the bash script approach and it worked.

1 Like
4

Hi @siv
Thanks for letting us know that your issue is resolved. It’s really helpful to other users to hear what worked.
Cheers,
Eddie