Versioning custom jsonschema

For engineers
1

Hi there,

I’d like to understand how to version custom jsonschemas if I introduce a breaking change.

Initial custom jsonschemas work out of the box – I create something like:

{
	"$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
	"description": "Example event",
	"self": {
		"vendor": "com.example",
		"name": "link_click",
		"format": "jsonschema",
		"version": "1-0-0"
	},

	"type": "object",
	"properties": {
		"elementId": {
			"type": "string"
		},
		"elementClasses": {
			"type": "array",
			"items": {
				"type": "string"
			}
		},
		"elementTarget": {
			"type": "string"
		},
		"targetUrl": {
			"type": "string",
			"minLength": 1
		}
	},
	"required": ["targetUrl"],
	"additionalProperties": false
}

igluctl lint and then igluctl static push to my self-hosted AWS Iglu server LB.

Now let’s say I make a breaking change, such as changing the elementId from type string to integer. I would expect that renaming/copying the file to 2-0-0 and updating the schema contents to the below would let me repeat the previous igluctl lint and igluctl static push steps (and add a new table, etc).

{
	"$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
	"description": "Example event",
	"self": {
		"vendor": "com.example",
		"name": "link_click",
		"format": "jsonschema",
		"version": "2-0-0"
	},

	"type": "object",
	"properties": {
		"elementId": {
			"type": "integer"
		},
		"elementClasses": {
			"type": "array",
			"items": {
				"type": "string"
			}
		},
		"elementTarget": {
			"type": "string"
		},
		"targetUrl": {
			"type": "string",
			"minLength": 1
		}
	},
	"required": ["targetUrl"],
	"additionalProperties": false
}

Instead, I see an error when I try to lint or push:

igluctl lint schemas/com.example/link_click/jsonschema/2-0-0 
Schema [com.example/link_click/jsonschema/2-0-0] contains a schema whose version is not 1-0-0
OK: com.example/link_click/jsonschema/2-0-0
TOTAL: 1 files corrupted
TOTAL: 1 valid schemas
TOTAL: 0 schemas didn't pass validation

Do I need to update my schema metadata reference in some way?

Thanks!

2

Hi @i.a.n, that error message is given when igluctl finds the 2-0-0 schema but it is missing he 1-0-0 schema. Igluctl tries to be strict about ensuring you have the complete sequence of schemas.

The way to fix this is to copy the original schema into a new file, but also keep the original file.

1 Like
3

To add a little more to the above from @istreeter - Iglu checks for “gaps” in schemas e.g., where 1-0-0 is missing or there are gaps between schema versions.

The main reason it takes these gaps into account is so that it can perform more graceful on the data destination side - as these are used to generate migration statements to run against the data warehouse.

1 Like
4

Thanks both @istreeter and @mike for the quick and thoughtful responses! That makes sense to me and is an easy fix :slight_smile:

1 Like