I have defined a context that I want to track for some but not all our custom events. Is there a way of specifying schemas of required contexts that need to be set for individual events in the event’s own jsonschema? I’d like events that are missing a required context to be rejected in the validation part.
Hi @boba
At the moment there isn’t a way of ensuring a custom context is attached to a specific event type. It’s an interesting idea, although it does create some complexity in that schemas become related to one another rather than standing individually.
If you are concerned about the context being missing from the event and if the data is required to understand the specific custom event, I’d suggest that the properties are added to the schema that specifies the event rather than trying to ensure the custom schema is always added.
1 Like
Hi Paul, thanks for your reply!
Is it possible to validate as an enrichment step whether all required contexts are provided for a certain event? We’d have a list of required contexts for each event schema ready. If we don’t use contexts, we’d have a lot of duplicate event property descriptions, making it harder to maintain our schemas if we change some entity in our app.
Alternatively, since we’re using Snowflake as our data warehouse (which supports JSON columns), I thought we can maybe send the contexts as objects in event properties? And attach the context schema to the property? Something like
“properties”: {
“user”: {
“description”: “xxx”,
“type”: “object”,
“schema”: “http://iglu-server/schemas/com.xxx.xxx/user_context/jsonschema/1-0-0”
}
}
Hey @boba, one idea worth exploring is to use the Javascript enrichment to run some custom functions that check for the context according to the logic you desire.
Throwing an exception would fail the event and it would go to bad rows - if you would like to avoid this, you could also have the JS enrichment attach a derived context to indicate the result of your checks. That way you could query that column in snowflake to identify rows that fail the tests & explore the root cause.
Adding custom validation rules for things like this use case is something we’ve discussed before, and is an idea that has generated some excitement internally.
It does involve some complex interactions with core components of the pipeline however, so at the moment we’re at the point of assessing the viability of the concept. So it’s hard to be any more specific than saying lots of us think it’s cool and hope we can figure out a viable path!
1 Like