Old Cached Schemas Enrich Error

For engineers Enrichment
1

Hi!

Sometimes we need to edit schema that is already in production.

For example: https://sp.napolke.ru/schemas/ru.napolke.sp/catalog_cardpreview_click/jsonschema/1-0-0

If we don’t edit schema everything is good. After editing schema we are getting this message.

:[{“level”:“error”,“message”:“error: Could not find schema with key iglu:ru.napolke.sp/catalog_cardpreview_click/jsonschema/1-0-0 in any repository, tried:\n level: "error"\n repositories: ["Local cached IGLU repository mirror [HTTP]","Iglu Client Embedded [embedded]","Custom schemas IGLU repository [HTTP]"]\n”}],“failure_tstamp”:“2020-09-07T11:34:17.706Z”}

Js example:
sp_nap(‘trackSelfDescribingEvent’, {
schema: ‘iglu:ru.napolke.sp/catalog_cardpreview_click/jsonschema/1-0-0’,
data: {
catalog_id: [‘asd_test’],
brand_id: [‘asd_test’]
}
});

We are using:
snowplow-stream-collector-kafka-0.14.0
and
snowplow-stream-enrich-kafka-0.19.1

I found this topic:

We tried to kill and restart enrichment process and it does’t work. We still getting this type of message.
We use containers. “container” is that upon restart, it starts with a clean history.

Is there any other way to clear cache or solve this problem in another way?

Thanks for the answer

1 Like
2

Hi @rodion_zhukov,

Currently, the only way to do this is to hard reboot enrich as you describe. Generally it’s ‘safe’ to do without risking data loss as long as you let the application drain.

In the general case, editing a schema after it has already been pushed to production is something we strongly discourage - ideally this is only done as an emergency measure. Principally because doing so affects the scalability and auditability of your data - but also it increases likelihood that you run into a pipeline-breaking issue (this second point will affect you less as a Kafka pipeline user for now, these are mainly associated with our loaders for database targets, but this may change).

I mention this because we have some tooling that might help avoid this scenario, which might be of interest:

Snowplow Mini is a small-scale single-instance version of the pipeline, which gives you real-time feedback. If the front-end devs and whoever creates schemas use them, generally most issues can be caught ahead of time.

Snowplow Micro is a tool we’ve built for use in CI/CD or locally - it can be set up to read your dev or production schemas, and run automated tests against them to validate for issues ahead of time. There are some guides to using it with common JS frameworks here and here.

I hope that’s useful!

3

Hi @Colm,

Ok i got you.
so i tried to make new schema and change version.
I made new schema (without deleting old one).
And it seems that nothing changed.

Does new schema equals to old one for snow plow?

old 1-0-1
new 1-0-99

“errors”:[{“level”:“error”,“message”:“error: Could not find schema with key iglu:ru.napolke.sp/catalog_cardpreview_click/jsonschema/1-0-99 in any repository, tried:\n level: “error”\n repositories: [“Local cached IGLU repository mirror [HTTP]”,“Iglu Client Embedded [embedded]”,“Custom schemas IGLU repository [HTTP]”]\n”}],“failure_tstamp”:“2020-09-07T13:13:16.149Z”}

4

Hi @rodion_zhukov,

That is slightly unexpected, the new schema should be a completely distinct asset, so once the new event is sent, the pipeline should check for the new schema.

It is possible that the failure response is cached if the new event is sent (with schema 1-0-99) before the schema exists. The other potential explanation I can think of is that the schema exists where it should, but something else is stopping Validation from being able to read it.

Let’s see if we can narrow it down:

Have you successfully sent other valid custom events through the pipeline?

Could you share your iglu_resolver.json configuration (with anything sensitive, like API keys, removed)?

5

@Colm,

I have created new schema and restarted enrich process.
And now it works.

It doesn’t work without restarting.

Yes we have already sent other events.
Do you need now iglu config? i can send it a bit later :slight_smile:

6

config:

{
“schema”: “iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-1”,
“data”: {
“cacheSize”: 500,
“repositories”: [
{
“name”: “Local cached IGLU repository mirror”,
“priority”: 2,
“vendorPrefixes”: [ “com.snowplowanalytics” ],
“connection”: {
“http”: {
“uri”: “http://localhost:9999/iglu/”
}
}
},
{
“name”: “Custom schemas IGLU repository”,
“priority”: 1,
“vendorPrefixes”: [ “ru.napolke.sp” ],
“connection”: {
“http”: {
“uri”: “http://localhost:8080/”
}
}
}
]
}
}

7

I’m a little confused as you shouldn’t have needed to restart enrich just for a new schema. But nothing looks amiss in your configuration - glad to hear you got it resolved!

1 Like
8

@Colm i! i found another problem :frowning:

As we discussed a bit earlier editing schemas is a bad practice.
So i made same schemas with another version.
For example:
old 1-0-1
new 1-0-6

Process of the problem:

  1. We made first schema (old)
  2. Sometime passed and we need to edit schema
  3. We edited schema (changed datatypes) and use old version (1-0-0)
  4. Snow plow said that he cant find schema. (Reloading snow plow didn’t change anything)
  5. As we discussed before i need to change schema version.
  6. We changed schema version to new one with new datatypes (1-0-7). And deleted old schema(1-0-0)
  7. I have already deployed new version and test everything. And it was ok after deploying and reloading sp. (collector was ok, enrich was fine etc)
  8. 2 Days passed and i have the same error :frowning: (We didn’t change anything) over 50% of edited customs events have this problem. But at start everything was fine

"errors":[{"level":"error","message":"error: Could not find schema with key iglu:ru.napolke.sp/catalog_card_click/jsonschema/1-0-7 in any repository, tried:\n level: \"error\"\n repositories: [\"Local cached IGLU repository mirror [HTTP]\",\"Iglu Client Embedded [embedded]\",\"Custom schemas IGLU repository [HTTP]\"]\n"}],"failure_tstamp":"2020-09-21T08:30:53.783Z"}

old schema

{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "description": "Napolke snowplow autogenerated schema",
  "self": {
    "vendor": "ru.napolke.sp",
    "name": "catalog_card_click",
    "format": "jsonschema",
    "version": "1-0-0"
  },
  "type": "object",
  "properties": {
    "catalog_id": {
      "type": "string"
    }
  },
  "additionalProperties": false
}

new schema

    {
      "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
      "description": "Napolke snowplow autogenerated schema",
      "self": {
        "vendor": "ru.napolke.sp",
        "name": "catalog_card_click",
        "format": "jsonschema",
        "version": "1-0-7"
      },
      "type": "object",
      "properties": {
        "sp_supplier_id": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "sp_category_id": {
          "type": "array",
          "items": {
            "type": "integer"
          }
        },
        "sp_subcategory_id": {
          "type": "array",
          "items": {
            "type": "integer"
          }
        },
        "sp_brand_id": {
          "type": "array",
          "items": {
            "type": "integer"
          }
        },
        "sp_catalog_id": {
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      },
      "additionalProperties": false
    }
9

And one more question.
Does same schema name and different versions are differents events for sp?

10

A few questions which might help diagnose the issue:

Are you using a static Iglu repo, or Iglu Server?
How are you uploading schemas? Do you use igluctl?
I notice you went from 1-0-0 to 1-0-7. Did you upload versions in between, or jump from 0 to 7?

Does same schema name and different versions are differents events for sp?

They will be different versions of the same event - so characteristics like event_name will be the same, however it will validate against the schema for the version specified.

11

Hi Colm!

Our backend developers will come back with the answers.

I found one interesting thing yesterday.
We have another event in production with the same trouble. And i decided to test deleting schema and creating the same. Event version for example (v 1-0-5). It has the same structure as “catalog_card_click” “1-0-7” from example above.
and event has same error.

If i delete json schema. And create same json schema with same version same name and same structure and then i wait about min for deploying schema. SP loads this events to enr_good topic without reloading enrich process:)

So i have several questions:

  1. I think sometimes server with schemas can send 404 response to snow plow enrich. And i think after first 404 response sp cashes answer and then it can’t find schema. Does snow plow caches 404 response?
  2. Can we send another answer for example 503?
12

Yes, the response from Iglu can get cached - you should be able to edit the cacheTtl setting in iglu resolver to configure the cache lifetime.

2 Likes
13

Do you mean this config?
I found only “cacheSize”: is this correct?
http://iglucentral.com/schemas/com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-1

14

i found )