BigQuery Loader 0.3.0 released

Announcements New releases
1

We have identified a security vulnerability in BigQuery Repeater in this version, which we’ve fixed in version 0.4.2. Please do not use this version and upgrade straight to 0.4.2. The upgrade notes below are still relevant in this regard.

We have released version 0.3.0 of BigQuery Loader, our family of apps that load Snowplow data into BigQuery.

This release adds support for adding labels to the Cloud Dataflow jobs that implement Loader and Forwarder. We’ve also carried over bugfixes to Forwarder from 0.1.1 release candidates which were missed in 0.2.0.

Many thanks to @the-fine for contributing the Forwarder bugfixes.

Support for GCP labels

Loader and Forwarder now accept an optional parameter labels, which can specify key-value pairs to be used as labels for the Dataflow job:

$ ./snowplow-bigquery-loader \
    --config=$CONFIG \
    --resolver=$RESOLVER
    --labels={"key1":"val1","key2":"val2"} # optional

$ ./snowplow-bigquery-forwarder \
    --config=$CONFIG \
    --resolver=$RESOLVER
    --labels={"key1":"val1","key2":"val2"} # optional
    --failedInsertsSub=$FAILED_INSERTS_SUB

This can be helpful if you are launching Loader as a Kubernetes job, because labels assigned to the job will not be replicated in the resulting Dataflow job.

Upgrading

This release comes very soon after 0.2.0, which brought significant changes. If you are upgrading from 0.1.x straight to 0.3.0, please refer to the BigQuery Loader 0.2.0 release post for some possible upgrade complexities and how to solve them.

For further details on changes, please see the release notes:

2 Likes