Event error with INVALID_DATA_PAYLOAD

For engineers
1

Hi Everyone

We have a snowplow setup with BigQuery Loader and in our mobile app.
We use Flutter Tracker to track and send the event to the collector endpoint.

We have a custom event setup on Iglu and is able to send the event to the endpoint and insert to BigQuery without issue.

But we found some failed event that was inserted into the dead letter bucket.
Example below

{
    "schema": "iglu:com.snowplowanalytics.snowplow.badrows/tracker_protocol_violations/jsonschema/1-0-1",
    "data": {
        "processor": {
            "artifact": "snowplow-enrich-pubsub",
            "version": "3.9.0"
        },
        "failure": {
            "timestamp": "2024-07-08T04:56:34.405806Z",
            "vendor": "com.snowplowanalytics.snowplow",
            "version": "tp2",
            "messages": [
                {
                    "json": {
                        "eid": "20e15d95-f848-4825-9149-472b3110a893",
                        "res": "1080x2408",
                        "tv": "andr-6.0.4 flutter-0.6.0",
                        "e": "ue",
                        "tna": "test-ns-qa",
                        "tz": "Asia/Bangkok",
                        "stm": "1720414593939",
                        "p": "mob",
                        "cx": "Encode payload",
                        "ue_px": "Encode payload",
                        "dtm": "1720414592531",
                        "lang": "\u0e44\u0e17\u0e22",
                        "aid": "com.example.qa"
                    },
                    "error": "INVALID_DATA_PAYLOAD"
                }
            ]
        },
        "payload": {
            "vendor": "com.snowplowanalytics.snowplow",
            "version": "tp2",
            "querystring": [],
            "contentType": "application/json; charset=utf-8",
            "body": "{\"eid\":\"20e15d95-f848-4825-9149-472b3110a893\",\"res\":\"1080x2408\",\"tv\":\"andr-6.0.4 flutter-0.6.0\",\"e\":\"ue\",\"tna\":\"test-ns-qa\",\"tz\":\"Asia\\/Bangkok\",\"stm\":\"1720414593939\",\"p\":\"mob\",\"cx\":\"Encoded payload\",\"ue_px\":\"Encoded payload\",\"dtm\":\"1720414592531\",\"lang\":\"\u0e44\u0e17\u0e22\",\"aid\":\"com.example.qa\"}",
            "collector": "ssc-3.0.1-pubsubsink",
            "encoding": "UTF-8",
            "hostname": null,
            "timestamp": "2024-07-08T04:56:33.873Z",
            "ipAddress": "",
            "useragent": "snowplow/andr-6.0.4 android/14",
            "refererUri": null,
            "headers": [
                "Host: example.qa.test.com",
                "user-agent: snowplow/andr-6.0.4 android/14",
                "content-type: application/json; charset=utf-8",
                "Content-Length: 56747",
                "accept-encoding: gzip",
                "X-Cloud-Trace-Context: ''",
                "Cookie: ''",
                "Via: 1.1 google",
                "X-Forwarded-For: '' , ''",
                "X-Forwarded-Proto: https",
                "Connection: Keep-Alive",
                "application/json; charset=utf-8"
            ],
            "networkUserId": ""
        }
    }
}

In the example above, we remove the ipAddress, cx, Cookie etc. but in the actual payload these data exist.
This is a custom schema we have setup and not the preset schema from snowplow.

The error only mentions INVALID_DATA_PAYLOAD and tracker_protocol_violations

Seem the payload violates some schema rule

And in this github
It only mentions that there is an issue with json self-describing

Weird thing we don’t understand is we try to resend this event with Python Tracker and it work we use the failed payload from the data failure key and send the same cx and ue_px and it can insert into BQ

We also try send the payload with postman and it also work.
So the issue is likely not from the payload

We also verify each field base in the github above on the payload section is not violate the rule set. the type is correct and the length is not over the limit

At first we through it some sort of length limit or size limit as the “Content-Length: 56747”, but we have send payload with way more data and way more length and is able to write to the BQ

We also tried to disable encoding in Flutter and send the payload still didn’t work and failed with the same error.

What we don’t understand is

  1. Why is the payload failed.
  2. If it failed due to json not self-describing why does send it via python SDK or postman work?

Anything we might have miss or any guidance on this matter would be very appreciated

We also try to look at the log from the enrich docker but it only output bad row without any detail on what is the error found.

2

Hi @Cha_le,

Could you try base64 decoding the cx and ue_px properties in the payload and seeing if it is valid JSON? Or disable base64 and check the JSON in co and ue properties.

I am wondering if there is some problem with JSON serialization where the something that is not serializable gets added to the payload.

Also I assume that the problem only occurs with this specific event/entity, other events that you track from the Flutter app work?

3

HI matus

Thank you for the answer.

I want to clarify one thing this event with the issue sometimes can write the data into our BQ but sometimes it failed with the above error from our flutter sdk.

The issue only occur with this custom event that we have found so far the rest of the event we have setup is working correctly.

The decode cx and ue_px are both are valid JSON and the disable base 64 are also valid JSON.

This event in the example has this schema
“product_evar”: {
“type”: “string”
},
“brandId”: {
“type”: “string”
},
“productdetail”: {
“type”: “string”
}

For our use case we store all the data, We want as a string and we will decode this later in BQ for example the data we track might look like this
{
“product_evar” : “;Chair multiple color;;1000.0;;evar33=example-brand|evar7=example-brand|evar51=5|evar54=normal,;”,
“brandId”: “Example-brand”,
“productdetail”:“[{“categorySlugs”:,“tagObjects”:,“byStore”:[{“sku”:“example-brand”,“type”:“configurable”,“stock”:{“qty”:0,“stockType”:“infinite”}}]},{“categorySlugs”:,“tagObjects”:,“byStore”:[{“sku”:“example-brand”,“type”:“configurable”,“stock”:{“qty”:0,“stockType”:“infinite”}}]},{“categorySlugs”:,“tagObjects”:,“byStore”:[{“sku”:“example-brand”,“type”:“configurable”,“stock”:{“qty”:0,“stockType”:“infinite”}}]}]”
}

The productdetail will be an appending list depend on what the product show and each can have different name and title.

What we notice is that it only failed for some product that we have. but we don’t find any special character that could cause it to failed. Also what thrown us off the most is that the event with the product that failed in flutter SDK.
Will work in python SDK and postman both can insert into BQ.

We notice that the flutter and python default encoding are a bit different as python use base64UrlEncode but flutter use base64Encode

But we try change the python encode to base64Encode same as flutter and try to send the event again the python event can insert into BQ.

We haven’t try to change the flutter to base64UrlEncode and send the event as this take some time to dig through the code.

So not really sure what could cause the issue.

4

Are you able to provide a code snippet (in Flutter) that replicates the issue of the failing event (with corresponding data)? This should give us a better ability to identify what hopefully sounds like an edge case.

5

Hi Mike

I will ask our Flutter developer if he can provide the code.
It might take some time as he is currently working on other projects. I will message you again once he has time to provide the code.

Thanks

1 Like
6

Hi

For this issue we end up with changing the payload itself and removing the issue field from the event.

Our Flutter developer doesn’t have a time slot to look further into this. We weren’t able to find out the root cause on this

Thank again for all the advice provide

7

No worries - if they do find themselves with some more time in the future we would more than welcome a bug report so we can look into it further!

1 Like