We are pleased to announce version 1.0.0 of sqs2kinesis
Sqs2kinesis an application used to protect a Snowplow pipeline on AWS during a sudden surge of events. The collector writes payloads to a SQS buffer instead of the standard kinesis sink, and sqs2kinesis then copies the events back into kinesis after the surge is over, or when the number of kinesis shards have scaled up.
New in this release:
- Better batching of kinesis
PutRecordsRequeststo avoid exceeding the limits of 5MB or 500 records per batch. - Writes bad events to the bad kinesis topic, if the sqs message cannot be base64 decoded
- Improved recovery after transient errors, or graceful failure and exit when necessary.
- Configuration file format has changed, to be more consistent with other snowplow applications.
- Many under-the-hood dependency upgrades
The application is now configured with a --config command line argument, which parses a hocon configuration file. This is a breaking change if you are upgrading from version 0.2.0, but it is all part of our wider plan to make Snowplow applications more uniform in how they are configured. Our docs site has a minimal example configuration file and a detailed reference for all configuration options.