I noticed some bad records in realtime pipeline in Elasticsearch. However when I looked into batch processing pipeline I noticed that EMR job just writes empty files in enriched.bad and shredded.bad buckets. It looks like this:
Hey @tyomo4ka - to be getting bads in your RT pipeline but not in batch suggests some difference between the two pipelines in terms of event validation or enrichment.
What messages are you seeing in the bads in your RT pipeline in Elasticsearch?