GCS Loader upgrade?

For engineers
5

Hey,

So i’ve tried running the gcs loader on appengine by creating a custom docker image for the same. The docker image runs the gcs loader bat file.

The docker image runs successfully locally and on appengine, it can acknowledge events from the pubsub but it doesnt write any file to the gcs bucket.

Could someone help me out?

The reason i’m trying to run it outside of a dataflow job is here.

Contents of dockerfile:

FROM openjdk:8-jdk-alpine

COPY snowplow-google-cloud-storage-loader-0.3.2/bin /bin
COPY snowplow-google-cloud-storage-loader-0.3.2/lib /lib
COPY script.sh script.sh

RUN apk update && apk add bash

CMD sh script.sh

Contents of script.sh:

./bin/snowplow-google-cloud-storage-loader \
--project=${PROJECT} \
--runner=DirectRunner \
--inputSubscription=${INPUT_PUBSUB_SUB} \
--outputDirectory=${GCS_BUCKET} \
--outputFilenamePrefix=output \
--shardTemplate=-W-P-SSSSS-of-NNNNN \
--outputFilenameSuffix=.txt \
--windowDuration=${WINDOW_DURATION} \
--compression=none \
--numShards=1 \
--dateFormat=YYYY/MM/dd/HH/

Thank you!