Redshift DDL for YAUAA context 1-0-3

For engineers Enrichment
1

Hi there!

We are using the YAUAA context 1-0-0 with this DDL: iglu-central/yauaa_context_1.sql at master · snowplow/iglu-central · GitHub.

Can someone confirm that I’d need to add those two following columns if I want to upgrade to the YAUAA context 1-0-3 (Enrich Kinesis 3):

-- added with YAUAA context 1-0-1 https://github.com/snowplow/iglu-central/issues/1077
operating_system_name_version_major VARCHAR(50) ENCODE ZSTD
operating_system_version_major VARCHAR(50) ENCODE ZSTD

?

Are there other columns required for YAUAA context 1-0-3 that are missing in the original DDL?

Thanks!

Edit: I just saw something weird. The last event in the table says it comes from YAUAA context 1-0-2

schema_vendor,schema_name,schema_format,schema_version,...
nl.basjes,yauaa_context,jsonschema,1-0-2,...

But we use the YAUAA context 1-0-0 in our configuration:

{
  "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/yauaa_enrichment_config/jsonschema/1-0-0",
  "data": {
    "name": "yauaa_enrichment_config",
    "vendor": "com.snowplowanalytics.snowplow.enrichments",
    "enabled": true
  }
}

Does Enrich Kinesis somehow automatically use the latest context it can? We use Enrich Kinesis 2.0.2 (which supports YAUAA context 1-0-2 I think) and Iglu Central as resolver.

If it already uses 1-0-2, then I guess the new columns are optional (we definitely don’t have operating_system_name_version_major and operating_system_version_major in the table but it works).

Edit2: oh I guess the versions of yauaa_enrichment_config and yauaa_context are totally unrelated. And so, even if yauaa_enrichment_config is in 1-0-0, Enrich Kinesis will still fetch the latest context that it supports (i.e. Enrich 2 → YAUAA 1-0-2, Enrich 3 → YAUAA 1-0-3). That would explain the events coming from YAUAA 1-0-2 in my table.

2

Hi Guillame,

As long as you’re using RDB loader R32 or higher (released in 2020 I think), you don’t need to run the DDL statement yourself - the loader’s automigration feature takes care of this for you.

But we use the YAUAA context 1-0-0 in our configuration:

The version 1-0-0 in the config you shared is actually the version number for the schema of the config itself, rather than the schema for the data that comes from the enrichment. I can see where it’s confusing, we have schemas for everything and it gets a bit meta (there’s even a schema for self-describing schemas :exploding_head: )

Does Enrich Kinesis somehow automatically use the latest context it can (we use Enrich Kinesis 2.0.2 so YAUAA context 1-0-2)? We use Iglu Central as resolver.

This part I stand to be corrected on the details as it’s not my particular area but I have confidence that the relevant information gives you what you need broadly. :slight_smile:

It doesn’t use the latest version automatically, but different versions of enrich will use different versions of the YAUAA API, and those will sometimes have different schemas. This is all hard-coded into enrich. So Enrich Kinesis 2.0.2 (or, more accurately common enrich v2+) uses YAUAA context schema 1-0-2, and YAUAA version 5.23.

It looks like it’s enrich v3.0.0+ that uses YAUAA context 1-0-3.

Note that all three YAUAA context versions are patch revisions, so there’s no breaking change across the three.

Edit: I missed your second edit but yes:

Edit2: oh I guess the versions of yauaa_enrichment_config and yauaa_context are totally unrelated. And so, even if yauaa_enrichment_config is in 1-0-0, Enrich Kinesis will still fetch the latest context that it supports (i.e. Enrich 2 → YAUAA 1-0-2, Enrich 3 → YAUAA 1-0-3). That would explain the events coming from YAUAA 1-0-2 in my table.

that’s exactly it!

2 Likes
3

Hi Colm,

Thanks for the thoughtful reply!

You clarified everything. I’m just going to keep the table as it is and upgrade to Enrich 3. :slight_smile:

2 Likes