Stream collector bad events to SQS and good events to Kinesis sinks

For engineers Collectors
1

Hi,

I’m wondering if it’s possible to have a Kinesis stream collector put good events into a Kinesis stream but bad events into SQS.

The reasoning behind this is that we only receive a couple of bad events per week, and running a Kinesis stream for this seems like overkill. It would also be cheaper just to have a queue in place of Kinesis.

I couldn’t find any way to make it work with the current setup tho – is there any workaround for this?

2

There isn’t at this stage internally - primarily because a lot of the tooling (e.g., the S3 loaders, streaming loaders etc) are designed to read off Kinesis rather than SQS.

It’s definitely cheaper to run on SQS, particularly if you have low bad volumes but a provisioned single shard Kinesis stream with 1 day retention should only be about $10 USD / month.

3

Thanks for the super fast response :slight_smile: