Invalid classification of SchemaIssues Errors

For engineers Enrichment
1

Hey,

I’m seeing some invalid SchemaIssue errors since uprading to stream enrich 5.2.0.

The error is as follows:

ValidationError with DataReports:2 SchemaIssues:0 for 'iglu:jet.cust/cx_basket/jsonschema/1-0-2'
dataReport -- Path:'$.total' Keyword:'multipleOf' Message:'$.total: must be multiple of 0.01' Targets:'0.01'
dataReport -- Path:'$.deliveryFee' Keyword:'multipleOf' Message:'$.deliveryFee: must be multiple of 0.01' Targets:'0.01'

This relates to the following context:

{"data":{"total":21.99,"id":"abc123","deliveryFee":2.99,"serviceFee":2},"schema":"iglu:jet.cust/cx_basket/jsonschema/1-0-2"}

An extract from the schema definition is as follows:

       "total": {
            "type": "number",
            "minimum": 0,
            "maximum": 360000,
            "multipleOf": 0.01
        }, 
        "deliveryFee": {
            "type": [
                   "number",
                   "null"
                    ],
            "minimum": 0,
            "maximum": 360000,
            "multipleOf": 0.01
        },

It wasn’t an issue in version 3.9.0 but has started occurring since the upgrade.

Thanks,
Rob

2

Are you able to by any chance share the original failed schema_violation JSON payload with any sensitive information redacted?

I’ve tried to recreate locally using Enrich 5.20 but I’m not able to recreate the error (with a 2.99 example value it passes validation for me).

3

Thanks for the quick response Mike! I’m not aware of any redaction on the logs as we have seen events fail this same schema checks in the past with more than 2dp’s.

I think that something may be amiss here though. This context is used in many applications but this error was only being reported in one. Since reverting the deployment, the errors stopped. I’ll investigate further.

4

Ah, to clarify I just meant manually redact any personal data you might have in the raw schema violation itself.

I had a double check of enrich - there is an upgraded version of the library we use for validation (from 1.0.70 to 1.0.76) but there are unit tests for multipleOf in that library so I’d be surprised if there was a regression there (but still possible!).

5

Hey @mike ,

We have managed to reproduce this and it does look like it’s due to the enricher was previously not decoding the base64 as it is now. I believe in 5.2.0 it is decoding correctly.

{"schema":"iglu:jet.cust\/cx_basket\/jsonschema\/1-0-3","data":{"serviceFee":2.6299999999999999,"id":"abc123","total":27.030000000000001,"type":"single","deliveryFee":0.48999999999999999}}

Interestingly decoding the base64 in JS truncates the decimal points but in shell it does not.

I think we’ll have to use a previous version. Correct the issues in production and then update one app adoption is nearing 100%.

Do you know the version where this change was introduced?

6

Are you able to share the full payload so that I can recreate this?

7

Here you go:

request

curl -H “Host: localhost:8080” -H “Cookie: sp=47eaaeb4-c39d-42b5-ba17-69082a03502e” -H “accept: text/html, application/x-www-form-urlencoded, text/plain, image/gif” -H “content-type: application/json; charset=utf-8” -H “user-agent: Darwin/24.3.0” -H “accept-language: en-GB,en;q=0.9” --data-binary “{"schema":"iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4","data":[{"tv":"ios-6.0.1","eid":"039C076F-207E-4B7B-B5A1-477F071FAAE9","dtm":"1738747617005","uid":"","stm":"1738747617010","tna":"customer_ios","ue_px":"eyJkYXRhIjp7ImRhdGEiOnt9LCJzY2hlbWEiOiJpZ2x1OmpldC5jdXN0XC9iYXNrZXRfYWRkXC9qc29uc2NoZW1hXC8xLTAtMCJ9LCJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvd1wvdW5zdHJ1Y3RfZXZlbnRcL2pzb25zY2hlbWFcLzEtMC0wIn0","p":"app","lang":"en-NL","res":"1179x2556","aid":"279120140337","vp":"1179x2556","cx":"eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvd1wvY29udGV4dHNcL2pzb25zY2hlbWFcLzEtMC0xIiwiZGF0YSI6W3siZGF0YSI6eyJlbnZpcm9ubWVudCI6ImRldiIsImN1cnJlbmN5IjoiR0JQIiwiY291bnRyeSI6InVrIiwibmFtZSI6Im9uZWFwcC1jdXN0b21lci1pb3MiLCJ2ZXJzaW9uIjoiMzUuNDcuMSIsImJ1aWxkIjoiMSIsImxhbmd1YWdlIjoiZW4ifSwic2NoZW1hIjoiaWdsdTpqZXRcL2N4X3BsYXRmb3JtXC9qc29uc2NoZW1hXC8xLTEtMCJ9LHsiZGF0YSI6eyJhbm9ueW1vdXNJZCI6IjU1QThFN0QyLUU2NzUtNDI5OC1CM0ZCLTNBQzU2RTFDM0E2MCIsInByZWZlcnJlZExhbmd1YWdlcyI6WyJlbi1OTCIsIm5sLU5MIl19LCJzY2hlbWEiOiJpZ2x1OmpldFwvY3hfdXNlclwvanNvbnNjaGVtYVwvMS0wLTQifSx7ImRhdGEiOnsic3RhdHVzIjpbImVzc2VudGlhbCIsImZ1bmN0aW9uYWwiLCJhbmFseXRpY2FsIiwicGVyc29uYWxpc2VkIl19LCJzY2hlbWEiOiJpZ2x1OmpldFwvY3hfY29uc2VudFwvanNvbnNjaGVtYVwvMS0xLTAifSx7InNjaGVtYSI6ImlnbHU6amV0XC9jeF9wcm9kdWNlclwvanNvbnNjaGVtYVwvMS0wLTEiLCJkYXRhIjp7InRlYW0iOiJjdXN0b21lci1yZXN0YXVyYW50LWFwcHMifX0seyJzY2hlbWEiOiJpZ2x1OmpldFwvY3hfYXBpXC9qc29uc2NoZW1hXC8xLTAtMCIsImRhdGEiOnsibmFtZSI6Ik1lbnVTaG93Y2FzZUFQSSIsImNvbnZlcnNhdGlvbklkIjoiYmJkNzNjNDMtYzcyYi00MGY0LWI0MDctZjEzNDIyZjcyNmMwIn19LHsic2NoZW1hIjoiaWdsdTpqZXQuY3VzdFwvY3hfcHJvZHVjdFwvanNvbnNjaGVtYVwvMS0wLTQiLCJkYXRhIjp7InByaWNlIjo4LjQxOTk5OTk5OTk5OTk5OTksImlkIjoiM2VkMjNmMDEtNzIzNS01MDc5LTgzNzktOTM4MGU2Y2EzNmY0IiwicXVhbnRpdHkiOjEsInBvc2l0aW9uIjoxfX0seyJzY2hlbWEiOiJpZ2x1OmpldC5jdXN0XC9jeF9iYXNrZXRcL2pzb25zY2hlbWFcLzEtMC0zIiwiZGF0YSI6eyJzZXJ2aWNlRmVlIjoyLjYyOTk5OTk5OTk5OTk5OTksImlkIjoiWXpWbU5ERXpNVGN0TkRrd01DMDBNVC12MSIsInRvdGFsIjoyNy4wMzAwMDAwMDAwMDAwMDEsInR5cGUiOiJzaW5nbGUiLCJkZWxpdmVyeUZlZSI6MC40ODk5OTk5OTk5OTk5OTk5OX19LHsic2NoZW1hIjoiaWdsdTpqZXRcL2N4X2NvbXBvbmVudFwvanNvbnNjaGVtYVwvMS0wLTEiLCJkYXRhIjp7Im5hbWUiOiJjb21wbGV4UHJvZHVjdE1vZGFsIiwidHlwZSI6ImRpYWxvZyJ9fSx7InNjaGVtYSI6ImlnbHU6amV0LmN1c3RcL2N4X2Z1bGZpbGxtZW50XC9qc29uc2NoZW1hXC8xLTAtMiIsImRhdGEiOnsibWV0aG9kIjoiZGVsaXZlcnkifX0seyJzY2hlbWEiOiJpZ2x1OmpldC5jdXN0XC9jeF9tZW51XC9qc29uc2NoZW1hXC8xLTAtMSIsImRhdGEiOnsiaWQiOiI4MzQ0ODAifX0seyJkYXRhIjp7InN0YXR1cyI6Im9wZW4iLCJtaW5pbXVtT3JkZXJWYWx1ZSI6MCwiaWQiOiIyMTc3MjkifSwic2NoZW1hIjoiaWdsdTpqZXQuY3VzdFwvY3hfcGFydG5lclwvanNvbnNjaGVtYVwvMS0wLTIifSx7InNjaGVtYSI6ImlnbHU6amV0LmN1c3QuZGlzY292ZXJ5XC9jeF9kaXNjb3ZlcnlDb21wb25lbnRcL2pzb25zY2hlbWFcLzEtMC0xIiwiZGF0YSI6eyJ0eXBlIjoiY2Fyb3VzZWwiLCJuYW1lIjoibWVudSBzaG93Y2FzZSJ9fSx7InNjaGVtYSI6ImlnbHU6Y29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93XC9tb2JpbGVfY29udGV4dFwvanNvbnNjaGVtYVwvMS0wLTMiLCJkYXRhIjp7Im9zVmVyc2lvbiI6IjE4LjEiLCJsb3dQb3dlck1vZGUiOmZhbHNlLCJpc1BvcnRyYWl0IjpmYWxzZSwicGh5c2ljYWxNZW1vcnkiOjE3MTc5ODY5MTg0LCJhcHBsZUlkZnYiOiI1NUE4RTdEMi1FNjc1LTQyOTgtQjNGQi0zQUM1NkUxQzNBNjAiLCJzY2FsZSI6MywicmVzb2x1dGlvbiI6IjExNzl4MjU1NiIsImRldmljZU1hbnVmYWN0dXJlciI6IkFwcGxlIEluYy4iLCJuZXR3b3JrVHlwZSI6IndpZmkiLCJsYW5ndWFnZSI6ImVuLU5MIiwiZGV2aWNlTW9kZWwiOiJpUGhvbmUxNiwxIiwib3NUeXBlIjoiaW9zIn19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3dcL2NsaWVudF9zZXNzaW9uXC9qc29uc2NoZW1hXC8xLTAtMiIsImRhdGEiOnsiZmlyc3RFdmVudElkIjoiMjg3RENCNUMtNTJFMS00QjU5LThDMjMtODVCOEFCOTZDOTZDIiwicHJldmlvdXNTZXNzaW9uSWQiOiI4ODA3YzgyNS1lMGJkLTRlYWQtYTA0Yi05OTdhZjE5YWI3ZWQiLCJmaXJzdEV2ZW50VGltZXN0YW1wIjoiMjAyNS0wMi0wNVQwOToyMDo1MC43OTZaIiwic2Vzc2lvbklkIjoiYzg4MzNlNDctOTgxZC00ZmI5LTk1NmEtYTRhM2I0NGJkNGZmIiwidXNlcklkIjoiNjNkNDZkZDctNjAxYi00NDMzLWFmZWYtNTM1NDRiYmY1OGQ5Iiwic2Vzc2lvbkluZGV4IjozLCJldmVudEluZGV4Ijo5MSwic3RvcmFnZU1lY2hhbmlzbSI6IkxPQ0FMX1NUT1JBR0UifX0seyJkYXRhIjp7ImlkIjoiRjZERkMyN0ItNTZEMy00MjlBLUE0MTctODRBNTI0RTZFMUNFIiwibmFtZSI6IlwvbWVudVwvaXRlbV9tb2RhbCJ9LCJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5tb2JpbGVcL3NjcmVlblwvanNvbnNjaGVtYVwvMS0wLTAifV19","tz":"Europe/Amsterdam","e":"ue"}]}” --compressed “https://localhost:8080/com.snowplowanalytics.snowplow/tp2”

8

If this is the raw data you are receiving my feeling is that there may potentially be some floating point rounding errors in the tracking code itself given that seems to be the raw event? I get basically the same payload in JS and Enrich (as a string) however if you parse this as a JSON object you will hit the precision limits of floating points in Javascript and as a result you will get a rounded result.

e.g.,

atob("eyJz...19")
>...cx_product\\/jsonschema\\/1-0-4","data":{"price":8.4199999999999999,"id":"3ed23f01-7235-5079-8379-9380e6ca36f4"...

JSON.parse(atob("eyJz...19"))
>..cx_product\\/jsonschema\\/1-0-4","data":{"price":8.42,"id":"3ed23f01-7235-5079-8379-9380e6ca36f4"...
9

Sorry, I should have mentioned this before but we are upgrading from 3.4.0 to 5.2.0.

We also switched the image which is when the error started being reported properly.
from: snowplow/stream-enrich-kinesis (no schema error detected)
to: snowplow/snowplow-enrich-kinesis (schema error detected)

There is nothing wrong with the way the enricher is currently validating. This was something that wasn’t being detected in stream-enrich-kinesis. In any case this is no longer a problem as of a long time ago. We’ll ensure we keep up to date on the latest releases going forward.

10

Ah fantastic - that’s great news, thanks for the update!