Inconsistent TSV format by enricher

1

Hello Snowplow Community,

I have been trying to load the enriched data into postgres. But the enricher writes the output with inconsistent tab spaces to the “enriched” kafka topic as following:

website web 2023-02-16 10:52:43.563 2023-02-16 10:52:43.553 2022-10-10 10:42:34.151 unstruct    5a305bf5-32fc-40d9-b0e8-66052f0f2c95        biz1    js-3.5.0    ssc-2.8.2-kafka snowplow-enrich-kafka-3.7.0     127.0.0.1       85094061-f702-4b62-a46d-20f7226b4741    29  99d70919-bbdb-499b-8cbf-f95613671444                                                https://snowplow.io/            https   snowplow.io 443 /                                                                                           {"schema":"iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0","data":{"schema":"iglu:com.xyz/node_events_public/jsonschema/1-0-0","data":{"name":"xyz"}}}                                                                          curl/7.79.1                     en-GB                                       1   30  693 1302                Europe/London           3440    1440    UTF-8   678 9015                                                2022-10-10 10:42:34.153             be9520e7-16a5-4d4e-afa1-8e269f99a1cf    2023-02-16 10:52:43.551 com.xyz  node_events_public  jsonschema  1-0-0       
Terminate batch job (Y/N)? Terminate batch job (Y/N)?

This file was generated by piping the output of a command line consumer to a file called 1.tsv through the following command.

.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic enriched > 1.tsv

Inconsistent format causes the snowplow-golang-sdk to throw errors when parsing this message. I have tried to test the pipeline e2e many times now, but the issue remains the same. Please guide me on how to resovle this issue or what are the probable causes behind this behaviour?

2

Hi @Gurankit_Pal_Singh , on a quick glance that enriched output looks OK to me. Please can explain more about why you think this has inconsistent tab spaces? And maybe share an example of the error you get from snowplow-golang-sdk?

The Snowplow enriched TSV format is explained here. There are 130 fields, each separated by a tab. If you see multiple tabs in a row (e.g. \t\t\t\t) then it means the fields they separate are null.

3

Yes, we resolved the issue, thanks for the reply.