We are pleased to announce version 1.5.0 of the Snowplow BigQuery Loader.
This release adds PubSub Flow Control mechanism to Streaming Loader, enabling greater control over memory consumption.
New config field maxRequestBytes
As part of Flow Control, we are introducing a new optional config field $.loader.consumerSettings.maxRequestBytes, the maximum size of unacked messages that stream loader can hold in memory at once. In combination with existing maxQueueSize field, the maximum number of unacked messages that stream loader can hold in memory at once, users are able to adjust memory consumption of their stream loader deployment.
Upgrading to version 1.5.0
Upgrading to version 1.5.0 is as simple as pulling the newest docker images:
docker pull snowplow/snowplow-bigquery-streamloader:1.5.0
docker pull snowplow/snowplow-bigquery-loader:1.5.0
docker pull snowplow/snowplow-bigquery-mutator:1.5.0
docker pull snowplow/snowplow-bigquery-repeater:1.5.0
Check out our docs site for full instructions on how to run and configure the BigQuery Loader.