Unknown Scala Collector Error

[anton]

Hi @pcb,

That seems to be a Kinesis outage. I’m bit surprised the collector crashed not with OOM though - this is something we’ve observed in the past, but allso 0.15.0 is a very old version (almost 2 years old). In recent version we’ve added an SQS buffer designed specifically to handle outages like that:

Stream Collector 2.0.0 released

In an effort to make Snowplow pipeline unbreakable, we are very excited to release Stream Collector 2.0.0 that adds surge protection for AWS. It is available on Docker Hub. While PubSub is auto-scalable, Kinesis is not, and when Kinesis can’t keep up with the throughput (in case of extreme traffic spikes for instance), events accumulate in collector memory for later (infinite) retries, which can lead to OutOfMemory errors. To address this potential issue, it is now possible to specify a SQS …

I recommend you to upgrade to the latest 2.1.0 version.