Hey Snowplowers,
I’m dealing with a legacy (soon-to-be-replaced) installation of the Snowplow batch pipeline and coming across an unusual issue with the Clojure collector.
We have around 300 KB events failing in each batch run (out of several GB total), with errors showing the offending lines are POST requests where the base64 payload is truncated about halfway through the data, obviously producing invalid JSON.
What’s strange is that, from the Tomcat logs, it looks like the log line for the request is actually being split across two Tomcat logfiles from Elastic Beanstalk; I have archived Tomcat logs that look like this:
-
var_log_tomcat8_rotated_localhost_access_log.2017-07-03.eu-west-1.i-000000000000.txt.gz:
2017-07-27 03:59:59 - - 12.345.67.89 POST 12.345.67.89 /com.snowplowanalytics.snowplow/tp2 200 - GuzzleHttp%2F6.2.1+curl%2F7.51.0+PHP%2F7.0.16 &cv=clj-1.0.0-tom-0.2.0&nuid=8dbdf709-c18c-4127-872f-c1dc617f3e28 - - - application%2Fjson eYxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxy -
var_log_tomcat8_rotated_localhost_access_log.2017-07-27-04.eu-west-1.i-000000000000.txt.gz:
xyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyx 2017-07-27 04:00:02 - - 12.345.67.89 POST 12.345.67.89 /com.snowplowanalytics.snowplow/tp2 200 - GuzzleHttp%2F6.2.1+curl%2F7.51.0+PHP%2F7.0.16 &cv=clj-1.0.0-tom-0.2.0&nuid=8dbdf709-c18c-4127-872f-c1dc617f3e28 - - - application%2Fjson eYabababababababa
I’m a bit baffled at this - a bug in Tomcat’s log rotation seems very unlikely but is the only explanation I can think of right now?!
I haven’t dug into the collector code yet but a problem there also seems unlikely given it’s hardly changed in 3 years!
Am I just incredibly unlucky? Any pointers v much appreciated…