Just started using the Stream Transformer image (snowplow/transformer-kinesis:3.0.0). At first it seems to be working fine and is outputting shredded files into the configured s3 prefix. However when looking at the SQS message that the transformer emits to downstream components like the RDB Loader, I noticed that the JSON message has the property data.count.good set to 0. According to the JSON schema iglu-central/2-0-0 at master · snowplow/iglu-central · GitHub This nummer indicates “Amount of good events in the batch”. This count seems to be wrong since it actually do put files in the location set in the property data.base
What could be the reason for this count of 0 and how to resolve it.
I’m using a static Iglu repository with schema list files generated by the igluctl 0.8.0 Release Version 0.8.0 · snowplow-incubator/igluctl · GitHub. I know that the documentation for Stream Transformer mention that Iglu server >= 0.6.0 is required, however no errors is logged when using the static repo with list files. Could this in some way be connected to the count=0 issue ?
Hey @BrianKjaerskov,
Even though count is zero, events should be transformed okay. This is a bug we’ve known for a time. We are planning to include the fix in 4.0.0.
Hi @enes_aldemir
Thanks for you quick reply. I guess I will need to wait for this bugfix before I will be able to use the RDB loader.
This bug shouldn’t affect the main workflow of RDB Loader. As I wrote in the previous message, events should be transformed without problem. Event count is mostly informational. Are you planning to use the event count in some other place ?
1 Like