Hi Everyone,
I am facing a weird issue where I see a higher number of events count in enrichment good topic compared to the collector good topic.
After debugging, I have realised that collector merges multiple events into an array of events before writing it to the good topic. While enrichment step is smart enough to split them and consider the atomic events separately.
Is it some sort of optimisation from the collector or an issue?
Thanks
Hey @Jayant_Kumar, it’s not an issue nor anything to worry about!
What you’re observing here is actually the batching settings of whichever tracker you’re using. If the collector receieves POST requests in batches, it’ll forward it in a batch as well.
By design the collector does as little as possible aside from the job of interacting with the client side (which is one of the things that makes the pipeline very loss-averse).
Enrich parses the batch and deals with events individually - and outputs in single events.
If, for example, you only used GET requests, you’d have a 1:1 map between [collector stream] : [good + bad streams].
Hope that helps explain it!