Collector 2.9.0 released

Announcements New releases
1

We are excited to release version 2.9.0 of the collector.

The primary reason for this release is to update the logic for /sink-health endpoint. It also comes with other nice improvements.

What’s new ?

Improved logic for /sink-health endpoint (#276)

Until this release, the collector was starting with /sink-health endpoint healthy, until proven unhealthy (which was a problem in case of Kinesis outage for instance). The new logic is to start unhealthy until proven healthy, either by successfully sending events to the queue or by successfully checking the stream with AWS/GCP API, whichever comes first.

More details can be found on the related issue.

Kinesis and SQS: MaxRetries configurable (#295)

MaxRetries is used in 2 scenarios:

  • In case of SQS collector or Kinesis collector without SQS buffer, /sink-health becomes unhealthy after this number of failed retries (even though it is retried forever).
  • In case of Kinesis collector with SQS buffer, this is the number of retries to Kinesis before switching to SQS.

It was hard coded to 3 and is now configurable (here for Kinesis and here for SQS).

Pub/Sub: Pub/Sub emulator support (#270)

It is now possible to use Pub/Sub emulator (e.g. for testing).

Using it is as simple as running the Docker image and setting PUBSUB_EMULATOR_HOST.

Pub/Sub: second layer of retry (#304)

There is now a second layer of retry for inserting records into Pub/Sub, in addition to the built-in retry mechanism of Pub/Sub publisher. When a record can’t be inserted after the built-in retries, it’s added to a buffer and every retryInterval, the collector retries to insert the failed inserts.

MaxBytes configurable (#272)

In case a collector payload is too big, a size violation bad row is emitted.

The maximum size for each type of queue was hard coded (1 MB for Kinesis, 192 KB for SQS, 10 MB for Pub/Sub 1 MB for Kafka).

It is now configurable (Kinesis, SQS, Pub/Sub, Kafka).

Upgrading to 2.9.0

If you are already using a recent version of the collector, then upgrading is as simple pulling the most recent docker image:

docker pull snowplow/scala-stream-collector-kinesis:2.9.0
docker pull snowplow/scala-stream-collector-pubsub:2.9.0
docker pull snowplow/scala-stream-collector-kafka:2.9.0

Check out our documentation pages for a full guide to setting up and running the collector.

3 Likes