Hello,
I am using the snowplow SDK for GoLang, but I am noticing something odd.
I am using the sdk version 0.1.0, and I noticed that it had a hardcoded property value count of 130.
I have noticed that my enricher is producing around 91. So I am wondering if there is a version compatibility issue that is not documented?
Has anyone else run into this issue?
Hey @hjaffan
Could you confirm what versions of the Snowplow Collector and Snowplow Enrich that you are running? (You should be able to see this metadata in your enriched event in the v_* fields).
I will check that, but in the meantime, do we know of any versions that are currently not supported with this SDK?
@hjaffan It’s difficult to pin down the exact version number when we last changed the number of fields in an enriched event, since we’ve refactored the codebase a few times, migrated repos, and changed how we version things since then - however I can confidently say it would be a long-deprecated version of the pipeline, I believe the last change like this would have been around 2015-2016.
It will be a lot less effort for us to look at the v_etl (and v_collector) value in your events to verify whether that’s a consideration, since we’d have to do a lot of digging to do it the other way around.
Just to make sure we’re on the same page/leave no stone unturned - not all 130 fields will be populated in the enriched event, which is a tab separated string (so it’s easy to miscount the empty ones) - does the go analytics SDK throw an error when you try to parse the event?
It does seem like it fails on this check:
func ParseEvent(event string) (ParsedEvent, error) {
record := strings.Split(event, "\t")
if len(record) != eventLength {
return nil, errors.New(fmt.Sprintf("Cannot parse tsv event - wrong number of fields provided: %v", len(record)))
}
return record, nil
}```
And that value is 131.
Let me check what the current events version is. But for now this does fail.Is the version we’re looking for stream-enrich-1.4.1-common-1.4.1?
That’s the version we’re after, which is up to date. Would it be possible to supply a TSV which fails? If you do, please replace any PII or other sensitive fields in it with dummy values.
Here’s an example:
"test\tweb\t2020-12-31 00:36:45.463\t2020-12-31 00:36:44.308\t2020-12-31 00:36:44.036\tpage_ping\t3017f65f-c0fb-4a13-bfa8-67a94f3e8aed\t\turjl9yoclvj\tjs-2.10.2\tssc-2.0.0-kinesis\tstream-enrich-1.4.1-common-1.4.1\t\txx.xxx.xxxx.xxx\t1431981418\t8d5536d3-3bd4-4a09-b8ae-3014f14b5738\t3\t127e8cb1-56a0-4068-be96-9e8bb9212e18\t\t\t\t\t\t\tfile:///Users/blah/Desktop/index.html\ttitle\t\tfile\t\t80\t/Users/blah/Desktop/index.html\t\t\t\t\t\t\t\t\t\t\t{\"schema\":\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0\",\"data\":[{\"schema\":\"iglu:com.google.analytics/cookies/jsonschema/1-0-0\",\"data\":{}},{\"schema\":\"iglu:com.snowplowanalytics.snowplow/web_page/jsonschema/1-0-0\",\"data\":{\"id\":\"0919e581-d1f5-4d57-b21f-618c9643055f\"}},{\"schema\":\"iglu:org.w3/PerformanceTiming/jsonschema/1-0-0\",\"data\":{\"navigationStart\":1609360983865,\"unloadEventStart\":0,\"unloadEventEnd\":0,\"redirectStart\":0,\"redirectEnd\":0,\"fetchStart\":1609360983872,\"domainLookupStart\":1609360983872,\"domainLookupEnd\":1609360983872,\"connectStart\":1609360983872,\"connectEnd\":1609360983872,\"secureConnectionStart\":0,\"requestStart\":1609360983872,\"responseStart\":1609360983872,\"responseEnd\":1609360985805,\"domLoading\":1609360985863,\"domInteractive\":1609360985959,\"domContentLoadedEventStart\":1609360985959,\"domContentLoadedEventEnd\":1609360985959,\"domComplete\":1609360986025,\"loadEventStart\":1609360986025,\"loadEventEnd\":1609360986025}}]}\t\t\t\t\t\t\t\t\t\t00\t0\t0\tMozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36\t\t\t\t\t\ten-US\t1\t0\t00\t0\t0\t0\t0\t0\t1\t30\t476\t821\t\t\t\tAmerica/Los_Angeles\t\t\t1440\t900\tUTF-8\t476\t821\t\t\t\t2020-12-31 00:36:44.049\t\t\t{\"schema\":\"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-1\",\"data\":[{\"schema\":\"iglu:com.snowplowanalytics.snowplow/ua_parser_context/jsonschema/1-0-0\",\"data\":{\"useragentFamily\":\"Chrome\",\"useragentMajor\":\"87\",\"useragentMinor\":\"0\",\"useragentPatch\":\"4280\",\"useragentVersion\":\"Chrome 87.0.4280\",\"osFamily\":\"Mac OS X\",\"osMajor\":\"11\",\"osMinor\":\"1\",\"osPatch\":\"0\",\"osPatchMinor\":null,\"osVersion\":\"Mac OS X 11.1.0\",\"deviceFamily\":\"Other\"}},{\"schema\":\"iglu:org.ietf/http_header/jsonschema/1-0-0\",\"data\":{\"name\":\"X-Forwarded-For\",\"value\":\"xx.xxx.xxx.xxx\"}},{\"schema\":\"iglu:org.ietf/http_header/jsonschema/1-0-0\",\"data\":{\"name\":\"Host\",\"value\":\"somethingplow.blah.com\"}},{\"schema\":\"iglu:org.ietf/http_header/jsonschema/1-0-0\",\"data\":{\"name\":\"User-Agent\",\"value\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36\"}},{\"schema\":\"iglu:org.ietf/http_header/jsonschema/1-0-0\",\"data\":{\"name\":\"Origin\",\"value\":\"null\"}},{\"schema\":\"iglu:com.dbip/location/jsonschema/1-0-0\",\"data\":{\"city\":{\"geoname_id\":xxxxxxx,\"names\":{\"de\":\"Somehwere\",\"en\":\"Somehwere\",\"fa\":\"Somehwere\",\"fr\":\"Somehwere\",\"ja\":\"Somehwere\",\"ko\":\"Somehwere\",\"pt-BR\":\"Somehwere\",\"ru\":\"Somehwere\",\"zh-CN\":\""```Has this event been truncated? The JSON (which looks like the derived_contexts field) isn’t valid (no closing brace) and it’s missing fields that typically appear after this. It also looks like it’s missing some tabs in some places (e.g., these zeros - for browser features should have a tab separator between them).
…36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36\t\t\t\t\t\tenUS\t1\t0\t00\t0\t0\t0\t0\t0\t1\t30\t476\t821\t\t\t\tAmeric
Sorry this could be from the copy and pasting that is happening between editors.
Could you provide me an example event to see if that works?
Sure - here’s an example from the Python Analytics SDK (not sure if there is a Go test with a full TSV in it).
angry-birds\tweb\t2017-01-26 00:01:25.292\t2013-11-26 00:02:05\t2013-11-26 00:03:57.885\tpage_view\tc6ef3124-b53a-4b13-a233-0088f79dcbcb\t41828\tcloudfront-1\tjs-2.1.0\tclj-tomcat-0.1.0\tserde-0.5.2\tjon.doe@email.com\t92.231.54.234\t2161814971\tbc2e92ec6c204a14\t3\tecdff4d0-9175-40ac-a8bb-325c49733607\tUS\tTX\tNew York\t94109\t37.443604\t-122.4124\tFlorida\tFDN Communications\tBouygues Telecom\tnuvox.net\tCable/DSL\thttp://www.snowplowanalytics.com\tOn Analytics\t\thttp\twww.snowplowanalytics.com\t80\t/product/index.html\tid=GTM-DLRG\t4-conclusion\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t{ "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0", "data": [ { "schema": "iglu:org.schema/WebPage/jsonschema/1-0-0", "data": { "genre": "blog", "inLanguage": "en-US", "datePublished": "2014-11-06T00:00:00Z", "author": "Fred Blundun", "breadcrumb": [ "blog", "releases" ], "keywords": [ "snowplow", "javascript", "tracker", "event" ] } }, { "schema": "iglu:org.w3/PerformanceTiming/jsonschema/1-0-0", "data": { "navigationStart": 1415358089861, "unloadEventStart": 1415358090270, "unloadEventEnd": 1415358090287, "redirectStart": 0, "redirectEnd": 0, "fetchStart": 1415358089870, "domainLookupStart": 1415358090102, "domainLookupEnd": 1415358090102, "connectStart": 1415358090103, "connectEnd": 1415358090183, "requestStart": 1415358090183, "responseStart": 1415358090265, "responseEnd": 1415358090265, "domLoading": 1415358090270, "domInteractive": 1415358090886, "domContentLoadedEventStart": 1415358090968, "domContentLoadedEventEnd": 1415358091309, "domComplete": 0, "loadEventStart": 0, "loadEventEnd": 0 } } ] }\t\t\t\t\t\t{ "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0", "data": { "schema": "iglu:com.snowplowanalytics.snowplow/link_click/jsonschema/1-0-1", "data": { "targetUrl": "http://www.example.com", "elementClasses": ["foreground"], "elementId": "exampleLink" } } }\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t1\t0\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t{ "schema": "iglu:com.snowplowanalytics.snowplow\\/contexts\\/jsonschema\\/1-0-1", "data": [ { "schema": "iglu:com.snowplowanalytics.snowplow\\/ua_parser_context\\/jsonschema\\/1-0-0", "data": { "useragentFamily": "IE", "useragentMajor": "7", "useragentMinor": "0", "useragentPatch": null, "useragentVersion": "IE 7.0", "osFamily": "Windows XP", "osMajor": null, "osMinor": null, "osPatch": null, "osPatchMinor": null, "osVersion": "Windows XP", "deviceFamily": "Other" } } ] }\t2b15e5c8-d3b1-11e4-b9d6-1681e6b88ec1\t2013-11-26 00:03:57.886\tcom.snowplowanalytics.snowplow\tlink_click\tjsonschema\t1-0-0\te3dbfa9cca0412c3d4052863cefb547f\t2013-11-26 00:03:57.886
not sure if there is a Go test with a full TSV in it
There are, but the TSV isn’t a copy-pastable string. This could be printed, but just using the example from the Python one should work!
stream-enrich-1.4.1-common-1.4.1
Just to provide confirmation, this version should be compatible with the Go SDK alright.