Good event with schema goes bad when using API Enricher with Snowplow Mini

Hi Snowplowers!

Found this strange behavior when using an API enrichment with Snowplow Mini 0.10.1.

  1. Created a valid schema that includes the following:
            "latitude": {
            "type": ["number","null"],
            "description": "The latitude",
            "minimum": -2147483647,              
            "maximum": 2147483647            
        },
        "longitude": {
            "type": ["number","null"],
            "description": "The longitude",
            "minimum": -2147483647,              
            "maximum": 2147483647   
            }
        }   
  1. I upload the schema to a clean Snowplow Mini.
  2. Test the schema with Postman sending a hit and I can see it in the good

  1. Configure the API enrichment to fetch the info from an API (response below) and it goes to bad, complaining that the latitude and longitude are strings when they are not.

This is the payload the API is responding (I send the same without the location_isp element when sending a hit from Postman directly to a webhook):

    {"location_isp": {"city": {"geoname_id": 5363943, "names": {"en": "La Jolla", "fa": "\u0644\u0627 \u0647\u0648\u06cc\u0627", "ja": "\u30e9\u30db\u30e4", "ko": "\ub77c\ud638\uc774\uc544", "ru": "\u041b\u0430-\u0425\u043e\u043b\u044c\u044f", "zh-CN": "\u62c9\u970d\u4e9a"}}, "continent": {"code": "NA", "geoname_id": 6255149, "names": {"de": "Nordamerika", "en": "North America", "es": "Norteam\u00e9rica", "fa": " \u0627\u0645\u0631\u06cc\u06a9\u0627\u06cc \u0634\u0645\u0627\u0644\u06cc", "fr": "Am\u00e9rique Du Nord", "ja": "\u5317\u30a2\u30e1\u30ea\u30ab\u5927\u9678", "ko": "\ubd81\uc544\uba54\ub9ac\uce74", "pt-BR": "Am\u00e9rica Do Norte", "ru": "\u0421\u0435\u0432\u0435\u0440\u043d\u0430\u044f \u0410\u043c\u0435\u0440\u0438\u043a\u0430", "zh-CN": "\u5317\u7f8e\u6d32"}}, "country": {"geoname_id": 6252001, "is_in_european_union": false, "iso_code": "US", "names": {"de": "Vereinigte Staaten von Amerika", "en": "United States", "es": "Estados Unidos de Am\u00e9rica (los)", "fa": "\u0627\u06cc\u0627\u0644\u0627\u062a \u0645\u062a\u062d\u062f\u0647\u0654 \u0627\u0645\u0631\u06cc\u06a9\u0627", "fr": "\u00c9tats-Unis", "ja": "\u30a2\u30e1\u30ea\u30ab\u5408\u8846\u56fd", "ko": "\ubbf8\uad6d", "pt-BR": "Estados Unidos", "ru": "\u0421\u0428\u0410", "zh-CN": "\u7f8e\u56fd"}}, "location": {"latitude": 32.8283, "longitude": -117.256, "time_zone": "America/Los_Angeles", "weather_code": "USCA0288"}, "postal": {"code": "92037"}, "subdivisions": [{"geoname_id": 5332921, "iso_code": "CA", "names": {"de": "Kalifornien", "en": "California", "es": "California", "fa": "\u06a9\u0627\u0644\u06cc\u0641\u0631\u0646\u06cc\u0627", "fr": "Californie", "ja": "\u30ab\u30ea\u30d5\u30a9\u30eb\u30cb\u30a2\u5dde", "ko": "\uce98\ub9ac\ud3ec\ub2c8\uc544 \uc8fc", "pt-BR": "Calif\u00f3rnia", "ru": "\u041a\u0430\u043b\u0438\u0444\u043e\u0440\u043d\u0438\u044f", "zh-CN": "\u52a0\u5229\u798f\u5c3c\u4e9a\u5dde"}}, {"geoname_id": 5391832, "names": {"en": "San Diego", "es": "Condado de San Diego", "fa": "\u0634\u0647\u0631\u0633\u062a\u0627\u0646 \u0633\u0646 \u062f\u06cc\u06af\u0648\u060c \u06a9\u0627\u0644\u06cc\u0641\u0631\u0646\u06cc\u0627", "fr": "Comt\u00e9 de San Diego", "ja": "\u30b5\u30f3\u30c7\u30a3\u30a8\u30b4\u90e1", "ko": "\uc0cc\ub514\uc5d0\uc774\uace0 \uad70", "pt-BR": "Condado de San Diego", "ru": "\u0421\u0430\u043d-\u0414\u0438\u0435\u0433\u043e", "zh-CN": "\u5723\u8fed\u6208\u53bf"}}], "traits": {"autonomous_system_number": 20001, "autonomous_system_organization": "Charter Communications Inc", "connection_type": "Corporate", "isp": "Charter Communications", "organization": "Spectrum"}}}

This is my API enrichment configuration

    {
  "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/api_request_enrichment_config/jsonschema/1-0-0",
  "data": {
    "enabled": true,
    "vendor": "com.snowplowanalytics.snowplow.enrichments",
    "name": "api_request_enrichment_config",
    "parameters": {
      "inputs": [
        {
          "key": "ipaddress",
          "pojo": {
            "field": "user_ipaddress"
          }
        }
      ],
      "api": {
        "http": {
          "method": "GET",
          "uri": "https://ssssssssss.execute-api.us-west-2.amazonaws.com/dev/?key=sssssssssssssss&ip={{ipaddress}}",
          "timeout": 15000,
	  "authentication": {
            "httpBasic": {
              "username": "anythingmandatory",
              "password": ""
            }
          }
        }
      },
      "outputs": [ 
        {
          "schema": "iglu:com.db-ip/location_isp/jsonschema/1-0-0",
          "json": {
            "jsonPath": "$.location_isp"
          }
        }        
      ],
      "cache": {
        "size": 1,
        "ttl": 30 
      }
    }
  }
}

I could change the schema to make latitude and longitude strings and the enrichment would work but I would rather not since I don’t know what’s causing this.

I haven’t tested on a production enricher … could this be tied to Mini? Any ideas what could be causing this, or how I can fix it?

Thank you
Joao Correia

2 Likes

Hi @joaocorreia,

We’ve stumbled upon similar and very confusing bug recently: https://github.com/snowplow/enrich/issues/374. The problem was deeply inside libraries we’re using - basically JSONPath lib extracts integers… not as integers, despite in the output still looks so.

On the bugtracker a similar case described (where integers turn into numbers or decimals, not strings) and it got fixed in recently released Enrich 1.4.0. I’m pretty sure that your case would also work with 1.4.0, but need to double-check that.

Thanks for the report @joaocorreia, I can confirm that this is a bug.

The root cause comes from a 3rd party library we use where fractional values are interpreted as string.

We’ve started working on Enrich 1.4.1 release and will apply a temporary patch until it is fixed upstream.

Thank you for the insight @oguzhanunlu @anton!

Will this fix also be applied to Snowplow Mini?

Thank you
Joao Correia

Of course @joaocorreia, as soon as Enrich 1.4.1 is ready, it’ll be followed by a Mini release.

Best regards,
Oguzhan

1 Like