How to remove and Update Schemas in Iglu Server

For engineers
1

Hello everyone!

We are running an Iglu Service in a K8s Cluster. We have a typically config.hocon file as usual:

    {
      "database": {
        "host": ${?IGLU_DB_HOST}
        "dbname": ${?IGLU_DB_NAME}
        "username": ${?IGLU_DB_USER}
        "password": ${?IGLU_DB_PASSWORD}
      }
      "repoServer": {
        "port" : "8080"
      }
      "superApiKey": ${?IGLU_SUPER_API_KEY}
    }

We publish the schemas on our Iglu Server using the --public option and IgluCTL like this:

$HOME/bin/${{ env.IGLU_FILE }} static push --public ${{ env.SCHEMAS }} ${{ env.IGLU_SERVER_K8_URI }} ${{ env.IGLU_SERVER_API_KEY }}

So now once we have the schemas uploaded to our iglu server, when i manually try to update one existing schema for example in Integration environment:

{"message":"Schema already exists"}

Im already aware that modifying an existing event schema can break a lot of things but at the end im the owner of the schemas and the iglu server and i want to force the patch/amend/update. How can i do that? Is there also a CURL sentence for this? Im using the Super API Key and also a read/write api key that i created for the specific vendor and its schema and it says the same message, Schema already exists.

I have also tried to Remove Schemas by executing this sentence:

curl HOST/api/schemas/VENDOR/SCHEMA_NAME/jsonschema/1-0-0 -X DELETE -H "apikey: API_KEY"

When i use the Super Api Key:
{“message”:“DELETE is forbidden on production registry”}

When i use a normal API Key:
{“message”:“Not enough permissions”}%

Checking your Code i can see here:
Link to Iglu Scala Code
I can notice this case when

case Permission.Super if patchesAllowed

How can i Toggle Patches? This is mainly my concern hehe
What config i need to enable on my iglu Server to be able to remove and update/Patch any kind of schema we want?
We know the downsides and problems of doing this, but there are certain situations that this is need, mainly in Integration environment. We have certain CI/CDs that send the schemas to Iglu, if someone merges something to a dev release branch and its broken, we need to create a new MAJOR:MINOR:PATCH version of the Schema and do a lot of work instead of fast fixing the Schema.

Thanks in advance!!!

1 Like
2

Hey @RaulCC at the same level as superApiKey you can set "patchesAllowed": true|false which will change the registry from Production to Development mode.

As an aside we also have a Helm Chart for running the Iglu Server that exposes this value for you: helm-charts/charts/snowplow-iglu-server/values.yaml at main · snowplow-devops/helm-charts · GitHub

3 Likes
3

Ey @josh , this worked like a Charm <3 solved the issue, thanks a lot for your fast response

1 Like
4

Sorry for disturbing again @josh , one fast question, is there any way to reduce the amount of time that the Snowplow Enricher service maintains the Schemas in its Cache? some magic parameter like:

"patchesAllowed": true

Based on this Post:

The cache will live as long as that Stream Enrich app instance is not terminated and restarted (e.g. by a server reboot)

5

Hey @RaulCC,

That’s quite an old post (8 years :sweat_smile:), these days you can configure the cache TTL in the resolver config as documented here: Iglu Resolver | Snowplow Documentation.

2 Likes
6

ooooooh I see, dunno how i missed that : ( thanks in advance!!! That was very helpful @stanch !!!