GCS Loader 0.3.2 template missing

For engineers GCP pipeline
1

Hey,

I was trying to deploy the latest version of the GCS Loader using the following command mentioned in the docs:

gcloud dataflow jobs run [JOB-NAME] \
  --gcs-location gs://sp-hosted-assets/4-storage/snowplow-google-cloud-storage-loader/0.3.1/SnowplowGoogleCloudStorageLoaderTemplate-0.3.2 \
  --parameters \
    inputSubscription=projects/[PROJECT]/subscriptions/[SUBSCRIPTION],\
    outputDirectory=gs://[BUCKET]/YYYY/MM/dd/HH/,\ # partitions by date
    outputFilenamePrefix=output,\ # optional
    shardTemplate=-W-P-SSSSS-of-NNNNN,\ # optional
    outputFilenameSuffix=.txt,\ # optional
    windowDuration=5,\ # optional, in minutes
    compression=none,\ # optional, gzip, bz2 or none
    numShards=1 # optional

But the template was not to be found in the given gcs location.

Could someone help me out?

Thank you!

2

Hey @siv ,

Documentation was outdated, we don’t use the templates any more, we now run the loader with a Docker image. We updated the instructions here. Sorry about that.

1 Like
3

Oh okay, thank you @BenB

1 Like