Custom schema events fall into bad-topic on Snpwplow+Kafka

For engineers
1

Hi,
can you please help we with the problem sending custom event to snoplow/kafka?
We have Snowplow opensource and Kafka topics on AWS.
I used this article Introductory guide to creating your own self-describing events and contexts [tutorial] to setup the Snowplow opensource and custom schema.

I created my schema and uploaded it to my AWS S3 Bucket. I have checked the schema with Igluctl lint and schema has passed the validation.
Then I have updated the resolver.json on my enrich server with the schema configuration.

I have setup Kafka “raw-topic” for my snowplow collector, “enriched-topic” for good messages after enrich server and “bad-2-topic” for bad messages after enrich server.
The problem is that all custom events are falling into “bad-2-topic” with the message below:

{
“schema”: “iglu:com.snowplowanalytics.snowplow.badrows/tracker_protocol_violations/jsonschema/1-0-0”,
“data”: {
“processor”: {
“artifact”: “snowplow-enrich-kafka”,
“version”: “3.7.0”
},
“failure”: {
“timestamp”: “2023-05-11T08:38:34.523165Z”,
“vendor”: “com.snowplowanalytics.snowplow”,
“version”: “tp2”,
“messages”: [
{
“schemaKey”: “iglu:com.example-company/onmouse/jsonschema/1-0-0”,
“schemaCriterion”: “iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-*”
}
]
},
“payload”: {
“vendor”: “com.snowplowanalytics.snowplow”,
“version”: “tp2”,
“querystring”: ,
“contentType”: “application/json”,
“body”: “{"schema":"iglu:com.example-company/onmouse/jsonschema/1-0-0", "data" : {\n\t"imgId": "event_id",\n\t"imgSrc": "2",\n\t"imgEvent": "smallImg"\n}}”,
“collector”: “ssc-2.8.2-kafka”,
“encoding”: “UTF-8”,
“hostname”: “MY-HOST.eu-central-1.elb.amazonaws.com”,
“timestamp”: “2023-05-11T08:38:29.510Z”,
“ipAddress”: “77.105.13.79”,
“useragent”: “python-requests/2.28.2”,
“refererUri”: null,
“headers”: [
“Timeout-Access: ”,
“X-Forwarded-For: 77.105.13.79, 165.225.201.152”,
“X-Forwarded-Proto: http”,
“X-Forwarded-Port: 80”,
“Host: MY-HOSt.eu-central-1.elb.amazonaws.com”,
“X-Amzn-Trace-Id: Root=1-645ca985-1b3cccd3252823c217353dcd”,
“User-Agent: python-requests/2.28.2”,
“Accept-Encoding: gzip, deflate”,
“Accept: /”,
“Cookie: _sp=305902ac-8d59-479c-ad4c-82d4a2e6bb9c”,
“application/json”
],
“networkUserId”: “aee02bca-c64a-459c-a627-fa6890424a70”
}
}

For some reason there is no error message, that would describe what is wrong.

My resolver json here

“schema”: “iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-3”,
“data”: {
“cacheSize”: 500,
“repositories”: [
{
“name”: “Iglu Central”,
“priority”: 0,
“vendorPrefixes”: [ “com.snowplowanalytics”],
“connection”: {
“http”: {
“uri”: “http://iglucentral.com”
}
}
},
{
“name”: “Custom Iglu Server”,
“priority”: 0,
“vendorPrefixes”: [ “com.snowplowanalytics”],
“connection”: {
“http”: {
“uri”: “http://MY-HOST.eu-central-1.elb.amazonaws.com/api”,
“apikey”: “86f2569f-54ce-47ae-aa6b-906520053a66”
}
}
},
{
“name”: “Iglu on S3 v2”,
“priority”: 2,
“vendorPrefixes”: [ “com.example-company” ],
“connection”: {
“http”: {
“uri”: “http://MY-BUCKET.s3-website.eu-central-1.amazonaws.com/”
}
}
}
]
}
}

My schema:

{
“$schema”: “http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#”,
“description”: “onmouse_img example”,
“self”: {
“vendor”: “com.example-company”,
“name”: “onmouse”,
“format”: “jsonschema”,
“version”: “1-0-0”
},
“type”: “object”,
“properties”: {
“imgId”: {
“type”: “string”,
“maxLength”: 50,
“description”: “…”
},
“imgSrc”: {
“type”: “string”,
“maxLength”: 50,
“description”: “…”
},
“imgEvent”: {
“enum”: [“smallImg”, “normalImg”],
“description”: “…”
}
},
“required”: [“imgId”, “imgSrc”, “imgEvent”],
“additionalProperties”: false
}

My data event:

curl http://MY-HOST.eu-central-1.elb.amazonaws.com:80/com.snowplowanalytics.snowplow/tp2 -H ‘Content-Type: application/json; charset=UTF-8’ -H ‘Cookie: _sp=305902ac-8d59-479c-ad4c-82d4a2e6bb9c’ --data-raw ‘{“schema”:“iglu:com.example-company/onmouse/jsonschema/1-0-0”, “data” : {“imgId”:“event_id”,“imgSrc”:“2”,“imgEvent”:“smallImg”}}’
curl http://MY-HOST.eu-central-1.elb.amazonaws.com:80/com.snowplowanalytics.snowplow/tp2 -H ‘Content-Type: application/json; charset=UTF-8’ -H ‘Cookie: _sp=305902ac-8d59-479c-ad4c-82d4a2e6bb9c’ --data-raw ‘{“schema”:“iglu:com.example-company/onmouse/jsonschema/1-0-0”, “data” : {“imgId”:“event_id”,“imgSrc”:“2”,“imgEvent”:“smallImg”}}’

For testing purposes I have put dummy attributes in my request and event fails with a proper message that such attribute doesnt exist in the schema. So that means that schema is available for snowplow enricher.
Can anyone help me how can I debug the solution and find why messages are failing?

2

Hi @apavel07,

I’m afraid the body of your request does not follow the format expected by the collector, which is more complex.

Instead of making raw curl requests to the collector, we recommend using one of our trackers – there is one for every major language, see here.

If you want to track an event from the console, you can make use of the tracking CLI app to fire events like this:

snowplow-tracking-cli --collector snowplow-collector.acme.com --appid myappid --method POST --schema iglu:com.snowplowanalytics.snowplow/event/jsonschema/1-0-0 --json "{\"hello\":\"world\"}"
3

As Matus has mentioned the easiest way is to use one of the existing trackers but if you do want to send a cURL request to the post endpoint you’ll need to make sure (for tp2) it is a POST request and the JSON data is wrapped in the Snowplow payload schema.

e.g.,

curl -XPOST http://MY-HOST.eu-central-1.elb.amazonaws.com:80/com.snowplowanalytics.snowplow/tp2 -H ‘Content-Type: application/json; charset=UTF-8’ -H ‘Cookie: _sp=305902ac-8d59-479c-ad4c-82d4a2e6bb9c’ --data-raw ‘{\"schema\":\"iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4\",\"data\":[{\"e\":\"pv\"}]}’

where the data property is an array of key value pairs corresponding to the tracker protocol. The Snowplow Chrome extension can do this automatically if you select an event and then use ‘Copy as cURL’.

image
image343×529 8.26 KB