What is the use case for Kinesis bad stream in scala collector..?

For engineers Collectors
1

I am aware of the fact that during enrichment, if there is a client side event which doesn’t comply with JSON schema then Enrich App routes that to a bad stream.(Please add to the answer if am missing any thing here)

But In the collector config we specify name of kinesis bad stream, But it is not documented, what is the use case to have a bad stream in collector.

Let me know, If I am not clear on my question …

2

Hi @devsaik, the Kinesis Collector has a bad stream to handle oversized events being sent to the collector. A single event cannot exceed 1MB in size and is defined in the code here.

I will make sure to get the documentation updated to reflect this!

3

Thanks a lot for clarifying.

4

@josh , could you please provide the details once more. the link mentioned in the response is giving 404.
What exactly is the purpose of it?

5

Hi @Shalini_Balakrishnan , I think the code he was refering to is here and the docs have been updating explaining what this config is for now as well Configure the Stream Collector | Snowplow Documentation

1 Like