In version 2.0.0 we introduced ‘surge protection’, the option to use an SQS buffer for persisting events when Kinesis is overwhelmed during traffic spikes. In this patch release we’ve overhauled the put retry logic for both Kinesis and SQS when surge protection is on.
Random glitches in Kinesis availability can trigger the SQS buffer, even when there is no meaningful increase in traffic. We wanted to give writing to Kinesis a fair chance before SQS is engaged.
This release is now available on Docker Hub.
For the full changes check out the release notes:
As with the previous release, the main goal of the surge protection feature is to ensure that Kinesis does not need to be needlessly overprovisioned to make it resilient to traffic spikes. You would still need to ensure that enough collector replicas are running to meet incoming traffic.