I’m trying to make use of the Scala SDK using
http://snowplowanalytics.com/blog/2016/03/23/snowplow-scala-analytics-sdk-0.1.0-released/#json-event-transformer as a guide but I’m not sure how to process bad events. The example code makes use of Scalaz Validation which is a little(!) over my head and is only filtering for successful events afaik.
Can anyone shine a little light or give an example?
As a side-note - the Scala Analytics SDK does not include support for parsing Snowplow bad rows currently. It’s a cool idea though - I’ve added a ticket, #13 Add support for bad rows.
Hey @christophe, it’s not quite I was after but I realise now my question was a little broad; I think you’re pointing to what to do about bad events in s3/kinesis from enrichment.
I’m actually trying to take the good enriched events and ship them off to somewhere else (BigQuery in this case). I’m from a Groovy background so Scala’s really new to me but the Scala SDK’s EventTransformer.transform(line) returns a ValidatedEvent which lead me to think that if the transform failed then those errors would be kept somewhere and I could at least log them.
So the docs say to extract the events you do something like:
val jsons = input.map(line => EventTransformer.transform(line)).filter(_.isSuccess).flatMap(_.toOption)
So I thought I’d be able to do something like:
val jsons = input.map(line => EventTransformer.transform(line)).filter(_.isFailure).flatMap(_.toOption)
To get the failed ones. But I’m not sure if I’m on the right track…
Hi @Shin - ah, I get you, you’re not talking about Snowplow bad rows, you’re asking about handling the failure side of the Scala Analytics SDK’s event transformer.
You are right - this
ValidatedEvent is built on top of Scalaz, specifically its
Validation applicative functor. This is old but probably still the best tutorial on
A Tale of 3 Nightclubs