Mobile Context to S3 Loader

For engineers Tracking SDKs
8

@Gaurav_Toshniwal what I find confusing here is that the JSON you’ve shared seems to be illegitimate as a Self-describing JSON. It doesn’t strike me as something the tracker can produce (at least, if the tracker was producing it I would have expected us to have a lot of bugreports about it).

Reformatting it to be more readable, here’s what you’ve shared:

{ schema=

{
  "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-1",
  "data": [
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-1",
      "data": {
        "sessionIndex": 2,
        "storageMechanism": "SQLITE",
        "firstEventId": "9cd0bb70-2561-4d3f-b0bd-8af4af365e63",
        "sessionId": "d8d4204c-2310-4735-87dc-4a3890e90411",
        "previousSessionId": "5a0cd403-5fcd-41ea-96d0-56d76b544b0b",
        "userId": "7e586e25-dbc5-479e-95a4-1461fa5af5d2"
      }
    },
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/mobile_context/jsonschema/1-0-1",
      "data": {
        "networkTechnology": "LTE",
        "carrier": "IND airtel",
        "osVersion": "9",
        "osType": "android",
        "androidIdfa": "6ad90f15-282d-4cfb-80d0-b84144f005e5",
        "deviceModel": "Redmi Note 7 Pro",
        "deviceManufacturer": "Xiaomi",
        "networkType": "mobile"
      }
    }
  ]
}
, data=null }

That should fail validation, and shouldn’t successfully come through the enrichment process. It should land in bad rows. A legitimate context array would be just this part:

{
  "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-1",
  "data": [
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-1",
      "data": {
        "sessionIndex": 2,
        "storageMechanism": "SQLITE",
        "firstEventId": "9cd0bb70-2561-4d3f-b0bd-8af4af365e63",
        "sessionId": "d8d4204c-2310-4735-87dc-4a3890e90411",
        "previousSessionId": "5a0cd403-5fcd-41ea-96d0-56d76b544b0b",
        "userId": "7e586e25-dbc5-479e-95a4-1461fa5af5d2"
      }
    },
    {
      "schema": "iglu:com.snowplowanalytics.snowplow/mobile_context/jsonschema/1-0-1",
      "data": {
        "networkTechnology": "LTE",
        "carrier": "IND airtel",
        "osVersion": "9",
        "osType": "android",
        "androidIdfa": "6ad90f15-282d-4cfb-80d0-b84144f005e5",
        "deviceModel": "Redmi Note 7 Pro",
        "deviceManufacturer": "Xiaomi",
        "networkType": "mobile"
      }
    }
  ]
}

ie. of the format {"schema": "iglu:com.snowplowanalytics.snowplow/client_session/jsonschema/1-0-1", "data": [{"schema": ..., "data": ...},{"schema": ..., "data": ...}]}

Indeed, {schema=..., data=...} is an invalid JSON.

I suspect Mike might be on to something in that there might be something about how you’re querying the data via Athena. There’s a tutorial on using Athena to query enriched data here - note that it’s slightly out of date, if your data is not in run= subfolders, than you’d need to remove PARTITIONED BY(run STRING) from the table definition.

Perhaps following that guide will unearth a different result - if it doesn’t, do let us know.