Hey! Thanks for the reply.
I discovered that I may still be using the old format of bad rows (?), although I upgraded my setup. I guess I must have missed something? Not sure what is going on but I’ve decided to move on, I’ll get back to bad rows a bit later.
My bad rows format looks like this:
{"line":"CwBkAAAACjMuODMu[......]C8xLTAtMAA=","errors":[{"level":"error","message":"Evaluating JavaScript script threw an exception: [org.mozilla.javascript.WrappedException: Wrapped java.lang.IllegalArgumentException: Input byte array has incorrect ending byte at 68 (user-defined-script#1)]"}],"failure_tstamp":"2022-01-05T13:14:21.290Z"}
I managed to run this:
but I’m failing to reprocess the resulting data so far.
I usually have .lzo files coming straight out of the collector/kinesis stream but this recovery process I just ran just dumped the value of line in the bad rows into files. So I now have files with one base64 event per line and I need to reprocess that. How to do that is still unclear to me, but I’m making progress. If you have any clue to help me, that would be amazing.
Regarding the more recent processes, I’ve tried:
- Spark
On AWS EMR, I wasn’t able to get the thing going. I was getting
Unexpected argument: com.snowplowanalytics.snowplow.event.recovery.Main
or, when I didn’t supply the above in MainClass, I was getting:
java.lang.NoClassDefFoundError: org/apache/spark/metrics/source/Source
Then I tried in a local Spark environment (although I’m new to this) but wasn’t able to setup the LZO compression happening. Running a lot into this when running the job:
GPL NativeCodeLoader: Could not load native gpl library
Then struggled way too long to try and get hadoop-lzo setup but I feel like it’s too old for the most recent Spark and Hadoop versions… or I’m just incompetent in Spark/Hadoop, which I am.
- Flink
Couldn’t get anything happen as I was consistently getting this message:
Ambiguous option/flag: --input
I tried versions 0.1.0, 0.2.0 and 0.3.1 of the snowplow-event-recovery jar for both setups, I was never able to make it work.
Now, du to this, I’m afraid that if I upgrade my Snowplow setup, I would not be able to recover bad rows anymore so it’s still important for me to understand how to make all of this work. I’ll probably come back to it soon and I’ll be able to share more details.
I’m curious if there’s a simple working example of bad rows reprocessing somewhere, like a bundle with dummy bad rows and the process to run against that. That would be a huge starting point for me. If it doesn’t exist, I’ll try to create my own and I’ll share it. No ETA on this though 