Our team set up a Near-real-time snowplow using ec2 as a collector and point to the Kinesis. When we set up a real-time snowplow, we lost data about 30% from batch data. My question is, where to start to debug when lost of data, for a start we check collector and from the log, we found out these errors
[scala-stream-collector-akka.actor.default-dispatcher-12737] WARN akka.actor.ActorSystemImpl - Illegal header: Illegal ‘user-agent’ header: Invalid input ‘/’, expected OWS, ‘EOI’, tchar, product-or-comment
or comment (line 1, column 178): Mozilla/5.0 (Linux; Android 6.0.1; SM-N9208 Build/MMB29K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/62.0.3202.84 Mobile Safari/537.36 SamsungBrowser/Cr
ossApp/0.1.88