ResolutionError ClientFailure Unexpected response

For engineers Enrichment
1

Hi

We have a snowplow setup on GCP using a terraform template.We have our own iglu server host on a VM. This VM is connect to a Load Balancer. We encountered issue where the event send was falling ResolutionError example error below

"failure": {
                "timestamp": "2024-10-10T10:58:09.479489Z",
                "messages": [
                    {
                        "schemaKey": "iglu:tourist/user-context/jsonschema/1-0-0",
                        "error": {
                            "error": "ResolutionError",
                            "lookupHistory": [
                                {
                                    "repository": "Iglu Central",
                                    "errors": [
                                        {
                                            "error": "NotFound"
                                        }
                                    ],
                                    "attempts": 11,
                                    "lastAttempt": "2024-10-10T10:58:06.389Z"
                                },
                                {
                                    "repository": "Iglu Central - Mirror 01",
                                    "errors": [
                                        {
                                            "error": "NotFound"
                                        }
                                    ],
                                    "attempts": 11,
                                    "lastAttempt": "2024-10-10T10:58:08.445Z"
                                },
                                {
                                    "repository": "Iglu Client Embedded",
                                    "errors": [
                                        {
                                            "error": "NotFound"
                                        }
                                    ],
                                    "attempts": 1,
                                    "lastAttempt": "2024-10-10T08:17:09.328Z"
                                },
                                {
                                    "repository": "Iglu Server",
                                    "errors": [
                                        {
                                            "error": "ClientFailure",
                                            "message": "Unexpected response: "
                                        }
                                    ],
                                    "attempts": 12,
                                    "lastAttempt": "2024-10-10T10:58:06.160Z"
                                }
                            ]
                        }
                    },
                    {
                        "schemaKey": "iglu:tourist/sign-up-email-start/jsonschema/1-0-0",
                        "error": {
                            "error": "ResolutionError",
                            "lookupHistory": [
                                {
                                    "repository": "Iglu Central",
                                    "errors": [
                                        {
                                            "error": "NotFound"
                                        }
                                    ],
                                    "attempts": 9,
                                    "lastAttempt": "2024-10-10T10:58:09.153Z"
                                },
                                {
                                    "repository": "Iglu Central - Mirror 01",
                                    "errors": [
                                        {
                                            "error": "NotFound"
                                        }
                                    ],
                                    "attempts": 9,
                                    "lastAttempt": "2024-10-10T10:58:09.469Z"
                                },
                                {
                                    "repository": "Iglu Client Embedded",
                                    "errors": [
                                        {
                                            "error": "NotFound"
                                        }
                                    ],
                                    "attempts": 1,
                                    "lastAttempt": "2024-10-10T08:17:09.893Z"
                                },
                                {
                                    "repository": "Iglu Server",
                                    "errors": [
                                        {
                                            "error": "ClientFailure",
                                            "message": "Unexpected response: "
                                        }
                                    ],
                                    "attempts": 9,
                                    "lastAttempt": "2024-10-10T10:58:08.923Z"
                                }
                            ]
                        }
                    }
                ]
            }

This failed in our Enrich pub sub

From what we understand from this document Understanding failed events | Snowplow Documentation

This error is due to the schema lookup failed to find the schema and it in the Iglu Server that we have setup.

But we are not really sure why it failed the message only mention Unexpected response

We have try calling our Iglu server using API mention in

We are able to call and get the schema list + detail and the health check is OK status

This is our iglu_resolver define in terraform

custom_iglu_resolvers = [
 {
      name            = "Iglu Server"
      priority        = 0
      uri             = "http://${module.iglu_lb.ip_address}/api"
      api_key         = random_uuid.iglu_super_api_key.result
      vendor_prefixes = []
    }
  ]

We are not sure what could be the issue any guidance or insight on what else we can further investigate would be appreciate

Thank you

2

So after debugging a bit more it turn out this is due to this setting in Terraform
redirect_http_to_https = true seem like the enrich send a HTTP not https to the iglu server and when it got redirect back from the iglu server lb it didn’t have a protocol to retry in https so result in the failed we got.

module "iglu_lb" {
  source  = "snowplow-devops/lb/google"
  version = "0.4.1"

  redirect_http_to_https  = true

  providers = {
    google = google
  }

After change it to false the schema is able to fetch again

3

Hi,

Additional info we have recently found. That still allows the redirect flag to be set to True and not encounter the error.

We made a mistake and referenced the Load Balancer IP in the Iglu resolver instead of the DNS name.

After change the setting to reference DNS instead it work without any error with the redirect flag = true

Hope this help.