Ensuring Non-Breaking Changes Within the Same Model Version in SchemaVer and Iglu Serve

1

Hello everyone,

I’m currently working with SchemaVer for versioning our JSON Schemas, as outlined in the Snowplow Analytics blog post on SchemaVer. I understand that SchemaVer uses the MODEL-REVISION-ADDITION format, where:

  • MODEL: Indicates a breaking change that prevents interaction with any historical data.
  • REVISION: Represents a change that may prevent interaction with some historical data.
  • ADDITION: Denotes a backward-compatible change that is compatible with all historical data.

My concern revolves around maintaining backward compatibility within the same model (major) version. Specifically, I’m wondering:

  • Does the Iglu Server provide any guarantees or enforcement mechanisms to prevent breaking changes within the same model version?
  • How can I ensure that schema evolutions within a model version remain non-breaking and adhere to the SchemaVer guidelines?

I’m looking for guidance on best practices to approach this issue. Any insights on how to leverage Iglu Server features or external tools to enforce schema compatibility would be greatly appreciated.

We are using BigQuery as our data warehouse.

Thank you in advance for your help!

2

Hi Hima - welcome to Discourse.

Which BigQuery loader version are you using?
Are you using the BDP console or interacting with Iglu Server directly?

The reason I ask is that one of the loaders uses a MODEL_REVISION_ADDITION naming convention for each column whereas the latest loader (to be more consistent with other loaders) uses just MODEL.

3

I am using the snowplow/bigquery-loader-pubsub:2.0.0-rc13-distroless
Which we do get the columns that use just MODEL which I do think is better for our downstream consumers.

I guess my question really is about this table and how we can enforce this.
i.e.
I want to make sure its not possible for someone to deploy
1.3.0 which has a breaking change from 1.2.0. Is this enforceable with any existing tooling or should I just build out some custom script to check based on this table.

Redshift Snowflake, BigQuery & Databricks
Add / remove / rename an optional field Non-breaking Non-breaking
Add / remove / rename a required field Breaking Breaking
Change a field from optional to required Breaking Breaking
Change a field from required to optional Breaking Non-breaking
Change the type of an existing field Breaking Breaking
Change the size of an existing field Non-breaking Non-breaking
4

At the moment it is enforced in the UI (when using the Data Structures builder) but it isn’t enforced in Iglu Server itself (in part because a small number changes to data can’t be determined by looking at the data structure itself) so the best option would be to use that table as a reference to determine if the change is breaking.

5

Thank you mike, that makes sense