Encoded bad rows in Elasticsearch - advanced debugging support

For data modelers & consumers
4

@jwhansome,

Indeed, the bad events generally take form

{ 
   line: "original raw event as a string record", 
   "errors":
      [{
         "level":"error",
         "message":"Here what's wrong with this event"
      }],
   "failure_tstamp":"timestamp"
}

This is true regardless of the collector you use though it would be base64 encoded in your case.

Unfortunately, we currently do not provide an ability to present that data decoded.

Here’s the tutorial describing how to debug the bad data in Elasticsearch using curl: Debugging bad rows in Elasticsearch using curl (without Kibana) [tutorial]. The approach is to filter out the events we do not care about (generated by bots, resulted due to OPTIONS requests, etc). The remaining would need to be examined to determine the reason for failure. It means decoding the value in line parameter (and fixing the underlying reason).

With regard to recovering data, if you are not using Lambda batch in your architechture - that is the “bad” events are not saved in S3 then I’m afraid you won’t be able to recover them. Having the events in S3 allows you to get them recovered and fed back to batch pipeline with the help of Hadoop Event Recovery. I think it’s reasonable taken the nature of “real-time” vs “batch” pipelines.