Snowplow Mini strange Schema Behavior

joaocorreia · November 5, 2020, 10:33pm

Hi Snowplowers,

I’m experiencing a weird behavior with this schema and the latest Snowplow mini, I wonder if someone can give me some insight on this.

I open the following browser tabs:

Consider this very simple schema: schemas/com.shopify/refund/jsonschema/1-0-0. (Discourse doesn’t let me attach a txt file)

{
"$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
"description": "Schema for Shopify refund events ",
"self": {
        "vendor": "com.shopify",
        "name": "refund",
        "format": "jsonschema",
        "version": "1-0-0"
},
"type": "object",
"properties": {
    "id": {
        "type": "integer",
        "description": "The refund id",
        "minimum": 0,              
        "maximum": 9223372036854775807
    },
    "order_id": {
        "type": "integer",
        "description": "The order id",
        "minimum": 0,              
        "maximum": 9223372036854775807
    },
    "created_at": {
        "description": "Cart event created at",
        "type": ["string","null"],
        "format": "date-time"
    },
    "note": {
        "description": "The refund note",
        "type": ["string","null"],
        "maxLength": 255
    },
    "user_id": {
        "type": ["integer","null"],
        "description": "The user id",
        "minimum": 0,
        "maximum": 9223372036854775807
    },
    "processed_at": {
            "description": "Cart event created at",
            "type": ["string","null"],
            "format": "date-time"
    },
    "restock": {
        "type": ["boolean","null"],
        "description": "Restock"
    },
    "total_duties_set": {
        "description": "The total duties set",
        "type": ["object","null"],        
        "properties": {
                        "shop_money": {
                        "description": "The total duties shop money amount",
                        "type": ["object","null"],        
                        "properties": {        
                                "amount": {
                                    "type": ["string","null"],
                                    "description": "Amount",
                                    "maxLength": 255
                                },
                                "currency_code": {
                                    "type": ["string","null"],
                                    "description": "Total duties shop money currency code",
                                    "maxLength": 5
                                }       
                            }
                        },
                        "presentment_money": {
                            "description": "The total duties presentment money",
                            "type": ["object","null"],        
                            "properties": {        
                                    "amount": {
                                        "type": ["string","null"],
                                        "description": "The total duties presentment money amount",
                                        "maxLength": 255
                                    },
                                    "currency_code": {
                                        "type": ["string","null"],
                                        "description": "The total duties presentment money currency code",
                                        "maxLength": 5
                                    }       
                                }
                        }  
                    }
    },
    "admin_graphql_api_id": {
        "description": "The refund note",
        "type": ["string","null"],
        "maxLength": 255
    }

},
"minProperties": 1,
"required": ["id","order_id"],
"additionalProperties": true
}

1 property minimum
Required properties of id and order_id
additionalProperties: true, meaning we can send more properties and the hit is still valid.

Reproduce:

I validate the schema with igluctl lint schemas/ and it says OK: com.shopify/refund/jsonschema/1-0-0
I upload this schema to Mini (and my Micro also reads local Iglu repo)
I verify the schema is uploaded to mini http://xxxxxxxxxxx:8081/api/schemas/com.shopify/refund/jsonschema/1-0-0
I create a postman POST hit to Snowplow mini https://mysnowplowmini.com/com.snowplowanalytics.snowplow/tp2 with following body:

{
“schema”: “iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4”,
“data”: [
{
“aid”: “shopify”,
“tna”: “spfy”,
“url”: “https://test.myshopify.com/refunds/create”,
“tv”: “2020-10”,
“e”: “ue”,
“se_ca”: “webhook”,
“se_ac”: “create”,
“se_la”: “refund”,
“p”: “app”,
“cs”: “UTF-8”,
“ue_px”:“ewogICJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvdW5zdHJ1Y3RfZXZlbnQvanNvbnNjaGVtYS8xLTAtMCIsCiAgImRhdGEiOiB7CiAgICAic2NoZW1hIjogImlnbHU6Y29tLnNob3BpZnkvcmVmdW5kL2pzb25zY2hlbWEvMS0wLTAiLAogICAgImRhdGEiOiB7CiAgICAgICJpZCI6IDY4MjE3OTMzMDIwNiwKICAgICAgIm9yZGVyX2lkIjogMjg2MjU2MDgzNzc5MCwKICAgICAgImNyZWF0ZWRfYXQiOiAiMjAyMC0xMS0wNFQxOToxOTo0NC0wNTowMCIsCiAgICAgICJub3RlIjogIiIsCiAgICAgICJ1c2VyX2lkIjogNjYxNDUxMjQ1MTAsCiAgICAgICJwcm9jZXNzZWRfYXQiOiAiMjAyMC0xMS0wNFQxOToxOTo0NC0wNTowMCIsCiAgICAgICJyZXN0b2NrIjogZmFsc2UsCiAgICAgICJkdXRpZXMiOiBbXSwKICAgICAgImFkbWluX2dyYXBocWxfYXBpX2lkIjogImdpZDovL3Nob3BpZnkvUmVmdW5kLzY4MjE3OTMzMDIwNiIsCiAgICAgICJyZWZ1bmRfbGluZV9pdGVtcyI6IFtdCiAgICB9CiAgfQp9”
}
]
}

If I execute the hit and go to Mini I can see the event on good in Kibana.

Now I add the following to my payload:

  "total_duties_set": {
    "shop_money": {
      "amount": "0.00",
      "currency_code": "USD"
    },
    "presentment_money": {
      "amount": "0.00",
      "currency_code": "USD"
    }
  },

This one (you will need to Base64 decode ue_px to see it) and when firing this to Snowplow Mini the hit doesn’t show up either in good or bad.

{
“schema”: “iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4”,
“data”: [
{
“aid”: “shopify”,
“tna”: “spfy”,
“url”: “https://test.myshopify.com/refunds/create”,
“tv”: “2020-10”,
“e”: “ue”,
“se_ca”: “webhook”,
“se_ac”: “create”,
“se_la”: “refund”,
“p”: “app”,
“cs”: “UTF-8”,
“ue_px”:“ewogICJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvdW5zdHJ1Y3RfZXZlbnQvanNvbnNjaGVtYS8xLTAtMCIsCiAgImRhdGEiOiB7CiAgICAic2NoZW1hIjogImlnbHU6Y29tLnNob3BpZnkvcmVmdW5kL2pzb25zY2hlbWEvMS0wLTAiLAogICAgImRhdGEiOiB7CiAgICAgICJpZCI6IDY4MjE3OTMzMDIwNiwKICAgICAgIm9yZGVyX2lkIjogMjg2MjU2MDgzNzc5MCwKICAgICAgImNyZWF0ZWRfYXQiOiAiMjAyMC0xMS0wNFQxOToxOTo0NC0wNTowMCIsCiAgICAgICJub3RlIjogIiIsCiAgICAgICJ1c2VyX2lkIjogNjYxNDUxMjQ1MTAsCiAgICAgICJwcm9jZXNzZWRfYXQiOiAiMjAyMC0xMS0wNFQxOToxOTo0NC0wNTowMCIsCiAgICAgICJyZXN0b2NrIjogZmFsc2UsCiAgICAgICJkdXRpZXMiOiBbXSwKICAgICAgInRvdGFsX2R1dGllc19zZXQiOiB7CiAgICAgICAgInNob3BfbW9uZXkiOiB7CiAgICAgICAgICAiYW1vdW50IjogIjAuMDAiLAogICAgICAgICAgImN1cnJlbmN5X2NvZGUiOiAiVVNEIgogICAgICAgIH0sCiAgICAgICAgInByZXNlbnRtZW50X21vbmV5IjogewogICAgICAgICAgImFtb3VudCI6ICIwLjAwIiwKICAgICAgICAgICJjdXJyZW5jeV9jb2RlIjogIlVTRCIKICAgICAgICB9CiAgICAgIH0sCiAgICAgICJhZG1pbl9ncmFwaHFsX2FwaV9pZCI6ICJnaWQ6Ly9zaG9waWZ5L1JlZnVuZC82ODIxNzkzMzAyMDYiLAogICAgICAicmVmdW5kX2xpbmVfaXRlbXMiOiBbXQogICAgfQogIH0KfQ”
}
]
}

Now the funny thing is both hits show up as good on Snowplow Micro.

{“total”:2,“good”:2,“bad”:0}

The problem seems to be this data and I cant seem to figure out why:

     "total_duties_set": {
    "shop_money": {
      "amount": "0.00",
      "currency_code": "USD"
    },
    "presentment_money": {
      "amount": "0.00",
      "currency_code": "USD"
    }
  },

What am I missing here?

Joao

Colm · November 6, 2020, 4:24pm

Hi Joao,

Thanks for reporting.

While we take a look at what might be happening here, I’d like to check if you’ve tried refreshing the kibana indexes?

Sometimes indexing problems can make it appear that data is missing, so would like to rule that out.

Best,

joaocorreia · November 6, 2020, 5:43pm

Hi @Colm,

I did refresh Kibana index, and just to be sure used a brand new instance of Mini 0.11.0.
I hoping this would be something obvious I’m missing …

Thank you
Joao

Colm · November 6, 2020, 6:00pm

I thought you might have alright - please bear with us while we take a look into it.

joaocorreia · November 6, 2020, 6:11pm

Thank you for looking into it @Colm!

oguzhanunlu · November 9, 2020, 4:21pm

Hey @joaocorreia , thanks for sharing a precise description of your issue, along with a step by step guide on how to reproduce.

I provisioned a Mini 0.11.0 and followed your steps. I’m able to see 2nd event in Kibana good index as well. I’d like to be on the same page regarding how we send events before digging further. Let me share how I send it so that you can retry and let us know if it is still the issue.

curl --location --request POST 'http://mini-address/com.snowplowanalytics.snowplow/tp2' \
--header 'Content-Type: application/json' \
--data-raw '{
  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",
  "data": [
    {
      "aid": "shopify",
      "tna": "spfy",
      "url": "https://test.myshopify.com/refunds/create",
      "tv": "2020-10",
      "e": "ue",
      "se_ca": "webhook",
      "se_ac": "create",
      "se_la": "refund",
      "p": "app",
      "cs": "UTF-8",
      "ue_px": "ewogICJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvdW5zdHJ1Y3RfZXZlbnQvanNvbnNjaGVtYS8xLTAtMCIsCiAgImRhdGEiOiB7CiAgICAic2NoZW1hIjogImlnbHU6Y29tLnNob3BpZnkvcmVmdW5kL2pzb25zY2hlbWEvMS0wLTAiLAogICAgImRhdGEiOiB7CiAgICAgICJpZCI6IDY4MjE3OTMzMDIwNiwKICAgICAgIm9yZGVyX2lkIjogMjg2MjU2MDgzNzc5MCwKICAgICAgImNyZWF0ZWRfYXQiOiAiMjAyMC0xMS0wNFQxOToxOTo0NC0wNTowMCIsCiAgICAgICJub3RlIjogIiIsCiAgICAgICJ1c2VyX2lkIjogNjYxNDUxMjQ1MTAsCiAgICAgICJwcm9jZXNzZWRfYXQiOiAiMjAyMC0xMS0wNFQxOToxOTo0NC0wNTowMCIsCiAgICAgICJyZXN0b2NrIjogZmFsc2UsCiAgICAgICJkdXRpZXMiOiBbXSwKICAgICAgInRvdGFsX2R1dGllc19zZXQiOiB7CiAgICAgICAgInNob3BfbW9uZXkiOiB7CiAgICAgICAgICAiYW1vdW50IjogIjAuMDAiLAogICAgICAgICAgImN1cnJlbmN5X2NvZGUiOiAiVVNEIgogICAgICAgIH0sCiAgICAgICAgInByZXNlbnRtZW50X21vbmV5IjogewogICAgICAgICAgImFtb3VudCI6ICIwLjAwIiwKICAgICAgICAgICJjdXJyZW5jeV9jb2RlIjogIlVTRCIKICAgICAgICB9CiAgICAgIH0sCiAgICAgICJhZG1pbl9ncmFwaHFsX2FwaV9pZCI6ICJnaWQ6Ly9zaG9waWZ5L1JlZnVuZC82ODIxNzkzMzAyMDYiLAogICAgICAicmVmdW5kX2xpbmVfaXRlbXMiOiBbXQogICAgfQogIH0KfQ"
    }
  ]
}'

On Micro showing 2 good events, it means the 2nd payload is also valid, given that Micro points to the same Iglu Server. This gives us the trust on 2nd payload as well.

I hope this is just a matter of how events are sent. Looking forward to hearing from you!

Kind regards,
Oguzhan

joaocorreia · November 9, 2020, 6:25pm

I’m using Postman, but I also tried curl and the result was the same. On micro both events are good, on mini, the second event don’t show neither in good or bad.

Are there any other ways to find whats happening inside Mini?

Thanks
Joao

joaocorreia · November 10, 2020, 7:58am

I reinstalled a brand new Snowplow Mini 0.11.0 and now it seems to work!?! Would it make sense to add a feature to remove all schemas and cache from Snowplow mini?

oguzhanunlu · November 10, 2020, 11:53am

Hey @joaocorreia,

I’m glad to hear that it is working now. It makes sense for sure and also sounds a bit dangerous. Let me share another way to achieve the same purpose.

Iglu Server’s config has a field that allows its’ users to overwrite or delete existing schemas. If you check the hyperlink, you’ll see that it is enabled by default on Snowplow Mini 0.11.0.

In this case, you can delete your schema by executing the following

curl --location --request DELETE 'http://mini-address/iglu-server/api/schemas/com.shopify/refund/jsonschema/1-0-0' \
--header 'apikey: your-super-api-key'

which should return { "message": "Schema deleted" }

If you want to double-check if it was deleted indeed, execute the following

curl --location --request GET 'http://mini-address/iglu-server/api/schemas/com.shopify/refund/jsonschema/1-0-0' \
--header 'apikey: your-super-api-key'

which should return { "message": "The schema is not found" }

I’m aware that this is not 100% what you asked for, this endpoint does not allow to delete all schemas at once, however it should make the cut, what do you think?

Kind regards,
Oguzhan

joaocorreia · November 10, 2020, 6:12pm

Thank you Oguzhan! This works!

Topic		Replies	Views
Good event with schema goes bad when using API Enricher with Snowplow Mini Enrichment	4	1869	October 26, 2020
Snowplow Mini - Custom Schemas Snowplow Mini	10	3171	April 14, 2016
Add Json Schema SelfDescribing Enrichment	3	1089	November 24, 2021
Trouble Using Custom Iglu Schemas for Snowplow Micro Troubleshooting	3	1522	June 10, 2022
Introducing Snowly, a local Snowplow request validator New releases	2	1510	June 4, 2016

Snowplow Mini strange Schema Behavior

Related topics