Custom schema NotFound (exists in Iglu and reachable)

For engineers
1

Hi, I’m using the enrich kinesis stream application version 3.7.0 and experiencing a bad event saying my custom schema does not exist (i.e. NotFound). The bad event record (real company name replaced with “privatecompany”):

{
    "error": {
        "error": "ResolutionError",
        "lookupHistory": [
            {
                "attempts": 100,
                "errors": [
                    {
                        "error": "NotFound"
                    }
                ],
                "lastAttempt": "2023-06-09T19:54:22.439Z",
                "repository": "Iglu Central"
            },
            {
                "attempts": 100,
                "errors": [
                    {
                        "error": "NotFound"
                    }
                ],
                "lastAttempt": "2023-06-09T19:56:23.722Z",
                "repository": "Iglu Central - Mirror 01"
            },
            {
                "attempts": 1,
                "errors": [
                    {
                        "error": "NotFound"
                    }
                ],
                "lastAttempt": "2023-06-08T14:23:24.029Z",
                "repository": "Iglu Client Embedded"
            },
            {
                "attempts": 100,
                "errors": [
                    {
                        "error": "NotFound"
                    }
                ],
                "lastAttempt": "2023-06-09T19:54:22.400Z",
                "repository": "Custom"
            }
        ]
    },
    "schemaKey": "iglu:com.privatecompany/dnb/jsonschema/1-0-0"
}

When I check the Iglu API for the list of schemas, this exact string is returned in the list and I’m able to GET the schema contents. I’ve also confirmed the table exists for the schema, which has no data. Other events whose schemas exist on Iglu Central are enriched fine, so I suspect this is an issue with the custom Iglu server.

My resolver configuration in DymamoDB is:

{
    "schema": "iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-2",
    "data": {
        "cacheSize": 500,
        "cacheTtl": 600,
        "repositories": [
            {
                "connection": {
                    "http": {
                        "uri": "http://iglucentral.com"
                    }
                },
                "name": "Iglu Central",
                "priority": 10,
                "vendorPrefixes": []
            },
            {
                "connection": {
                    "http": {
                        "uri": "http://mirror01.iglucentral.com"
                    }
                },
                "name": "Iglu Central - Mirror 01",
                "priority": 20,
                "vendorPrefixes": []
            },
            {
                "connection": {
                    "http": {
                        "apikey": "<obfuscated>",
                        "uri": "https://sp-iglu-dev.privatecompany.com"
                    }
                },
                "name": "Custom",
                "priority": 0,
                "vendorPrefixes": [
                    "com.privatecompany"
                ]
            }
        ]
    }
}

Are there tips on what else I could check?

2

Also, I just found I was missing /api at the end of my custom Iglu server in the resolver config from the example on Iglu Resolver | Snowplow Documentation. Waiting to see if that resolves the issue…

1 Like
3

Did that resolve it for you @pt-mike ? Was going to be my suggestion as well to add the /api suffix.

4

@josh thanks for that ping… I’ve been testing on and off today but I believe I still get the NotFound. I was wondering if https is also supported? I’m trying now with http:// for the Iglu server.

5

@josh , well I did some debugging from the Iglu server port using tcpdump and I can verify the enricher is obtaining my schema just fine with an http 200 over http (not https). I suspect maybe that was the issue, but that doesn’t make sense because the AWS ALB is terminating the SSL, unless there’s a limitation in the SDKs used by Iglu clients.

Next step is for me to check the kinesis transformer, but I have to wait on a PR review. More to come tomorrow…

6

Hey @pt-mike so we setup out Iglu Servers with ALBs and terminate TLS there as well - and I can 100% confirm that that does indeed work! I assume you don’t want to share your Iglu Server endpoint publicly but I would hazard a guess that there is something not quite right with your HTTPS setup at the Load Balancer end.

Have you got particularly strict cypher’s allowed for TLS? Are you using SNI?

7

Hey @josh, we found the issue. It was a couple of things. First a stale API key being used by the RDB loader, and second the iglu_resolver.json wasn’t even referencing an API key along with some confusion on naming (api_key vs. apikey). So a few things at play, but everyting is working as expected.

tcpdump came in real handy to track this down. I could see enricher and the transformer were using API keys, but not the RDB loader…

8

Glad to hear you got it resolved and working!

1 Like