We are pleased to announce version 1.6.0 of the Snowplow BigQuery Loader.
This release introduces a cache. We compared 1.6.0 against the latest stable version under high load and found that the cached version was able to load events faster than the control.
More details about the cache
Each time an event is loaded into BigQuery in the stream loader, that event will conform to one of a limited number of schemas defined as valid for your pipeline.
We identified an operation that was occurring each time that we loaded an event into BigQuery that only needed to happen the first time each schema is loaded. So, we introduced a cache for that operation.
Upgrading to version 1.6.0
Upgrading to version 1.6.0 is simple as pulling the newest docker images:
docker pull snowplow/snowplow-bigquery-streamloader:1.6.0
docker pull snowplow/snowplow-bigquery-loader:1.6.0
docker pull snowplow/snowplow-bigquery-mutator:1.6.0
docker pull snowplow/snowplow-bigquery-repeater:1.6.0
Check out our docs site for full instructions on how to run and configure the BigQuery loader.