I’ve got the snowplow collector, enrichment and kinesis stream working but I can’t figure out how to get rid of the Thrift data stored in the raw kinesis stream and it’s stopping me from being to take this payload and pass it to other systems.
In the image below, you can see the serialised data at the top with the encoding characters, but I do not want to have it in my kinesis.records.data JSON, but I can’t seem to figure out how to remove it. All I would like is the properly formatted JSON below it
Hi @TimLavelle generally you want to be passing the “enriched” data not the “raw” data of on to other systems as this is not only in a simpler format (TSV) but also several analytics SDKs available in different languages for working with this data. It has also by this point been validated and enriched so you know that you are working with good data!
Is there a reason you want to pass the “raw” data onto other systems? For archiving and backup we recommend setting up the Snowplow S3 Loader which can consume and bundle up this thrift format onto S3.
Hey Josh - thanks for the reply, and you’re right… I do want to pass the enrich data and I’ve gotten a bit closer as now my enricher is working…
But the enrichment data now shows in the logs as space delimited versus a JSON object, so I’m trying to debug that and have gotten close still, but not there yet…
Using the below, I can start to get the a JSON ouput, but the data looks to be base64 still?
event.Records.forEach(function(record) {
// Kinesis data is base64 encoded so decode here
var data = Buffer.from(record.kinesis.data, 'base64');
console.log(JSON.stringify(data));
});
Buffer in JS needs conversion to String if you want it to be a bit human readable
And this is where I’m getting stuck… I’ve tried numerous ways to turn the buffer into a string, then the string into a JSON object but can’t seem to figure this out. I’m no newbie to JS, but this is throwing my through a hoop.
Is there a blog post or previous discourse that talks about using JS to convert the buffer?
Been looking at TSV to JSON online and can see some examples, but when i run the TSV output in an online linter, it still doesn’t produce the JSON I’m looking for
[
'2/10/2020, 12:34:58 PM',
'\tweb\t2020-02-10 12:34:57.961\t2020-02-10 12:34:56.774\t2020-02-10 12:34:56.686\tpage_view\t06093cea-0fce-49f4-bda5-395bb4ea04dd\t\tcf\tjs-2.13.0\tssc-1.0.0-kinesis\tstream-enrich-1.0.0-common-1.0.0\t\t192.168.43.111\t\t4d3186b0-ead1-409f-ad10-bc08708b924d\t2\t163d1f8b-e73d-4982-86c5-0fd00fdf80a9\t\t\t\t\t\t\t\t\t\t\t\thttp://px.system/\tCambodia Vision b\u0000\u0013 Patient System b\u0000\u0013 The gift of sight\t\thttp\tpx.system\t80\t/\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36\t\t\t\t\t\ten-AU\t1\t0\t0\t0\t0\t0\t0\t0\t0\t1\t24\t1680\t939\t\t\t\tAustralia/Sydney\t\t\t1680\t1050\tUTF-8\t1680\t1653\t\t\t\t\t\t\t\t\t\t\t\t2020-02-10 12:34:56.688\t\t\t\t5979bf2e-5537-4c06-a063-3dd9ecfa8f44\t2020-02-10 12:34:56.772\tcom.snowplowanalytics.snowplow\tpage_view\tjsonschema\t1-0-0\t\t'
]
Gets output to:
[
{
"[": " '2/10/2020, 12:34:58 PM',"
},
{
"[": " '\\tweb\\t2020-02-10 12:34:57.961\\t2020-02-10 12:34:56.774\\t2020-02-10 12:34:56.686\\tpage_view\\t06093cea-0fce-49f4-bda5-395bb4ea04dd\\t\\tcf\\tjs-2.13.0\\tssc-1.0.0-kinesis\\tstream-enrich-1.0.0-common-1.0.0\\t\\t192.168.43.111\\t\\t4d3186b0-ead1-409f-ad10-bc08708b924d\\t2\\t163d1f8b-e73d-4982-86c5-0fd00fdf80a9\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\thttp://px.system/\\tCambodia Vision b\\u0000\\u0013 Patient System b\\u0000\\u0013 The gift of sight\\t\\thttp\\tpx.system\\t80\\t/\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\tMozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36\\t\\t\\t\\t\\t\\ten-AU\\t1\\t0\\t0\\t0\\t0\\t0\\t0\\t0\\t0\\t1\\t24\\t1680\\t939\\t\\t\\t\\tAustralia/Sydney\\t\\t\\t1680\\t1050\\tUTF-8\\t1680\\t1653\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t2020-02-10 12:34:56.688\\t\\t\\t\\t5979bf2e-5537-4c06-a063-3dd9ecfa8f44\\t2020-02-10 12:34:56.772\\tcom.snowplowanalytics.snowplow\\tpage_view\\tjsonschema\\t1-0-0\\t\\t'"
},
{
"[": "]"
}
]
True! We have analytics SDKs, which you can use to transform form an enriched event tsv to a more amenable JSON.
The recommended approach would be to consume the enriched stream, use one of the SDK’s (in your case the JS one) to transform the data to JSON, and then carry on with your application logic safe in the knowledge that everything that hits the enriched stream has already been validated and will always be of the same format.
As others have said, if you need to consume the raw stream directly, that’ll be a lot more of a headache!
Guys, love how quickly you all helped and came to the aid, such a fantastic group here!
I read the SDKs last night, but at 12:30am I think my mind was a bit bleh and I didn’t find the JS SDK…
Woke up this morning, had much needed coffee(s) and went back to the SDK docks and boom, saw the JS SDk, placed it in my app and updated the lambda function and low and behold, a beautifully formatted JSON object.
Have to do a bit more tweaking, but am very pleased for the help and thank you again!
