Custom API enrichment not being hit

For engineers Enrichment
1

I’m trying to get a custom API Enrichment working ( Custom API Request enrichment | Snowplow Documentation ) for stream-enrich-kafka (docker image tag snowplow/stream-enrich-kafka:2.0.5) but am unable to get even the simplest example to work. My config is:

{
    "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/api_request_enrichment_config/jsonschema/1-0-0",
    "data": {
      "enabled": true,
      "vendor": "com.surfside.user",
      "name": "sample_api_enrichment_config",
      "parameters": {
        "inputs": [
            {
                "key": "auctionId",
                "json": {
                    "field": "unstruct_event",
                    "schemaCriterion": "iglu:io.surfside.advertising/auction_init/jsonschema/1-0-0",
                    "jsonPath": "$.auctionId"
                }
            }
        ],
        "api": {
          "http": {
            "method": "GET",
            "uri": "http://host.docker.internal:4001/api?email={{auctionId}}",
            "timeout": 15000,
            "authentication": {}
          }
        },
        "outputs": [ 
          {
            "schema": "iglu:io.surfside.advertising/auction_enrich/jsonschema/1-0-0",
            "json": {
              "jsonPath": "$.auctionId"
            }
          }        
        ],
        "cache": {
          "size": 200,
          "ttl": 6000
        }
      }
    }
  }

The http service I’m targeting is a simple express app that just repeats the input (auctionID) back out. It also logs whenever it is hit, and I’m not seeing any log messages

Things I have tried:

  1. Confirming my config file has validated
  2. Confirming that my jsonschema are reachable by the resolver
  3. Confirming that my auction_init events are coming through (I have a tool which watches the kafka stream for the schema criteria I target)
  4. Confirming that host.docker.internal:4001 is reachable by the enricher container

I can’t seem to get the enrichment to work. I assume that stream-enrich-kafka supports the feature since I get validation errors if the config is incorrect

2

@ShortRoundDev, the name of your enrichment configuration is not correct. It has to be “api_request_enrichment_config” but you changed it to “sample_api_enrichment_config”. As a result, the configuration cannot be validated against its own JSON schema, iglu-central/1-0-0 at master · snowplow/iglu-central · GitHub.

3

I was under the impression that the “name” field was the name of the enrichment, and not the name of snowplow’s enrichment schema?

4

What makes Snowplow special is the ability to create data where everything is validated before the data can reach its destination. The enrichment configurations themselves need to be validated against the corresponding JSON schemas. To be able to validate, we use so-called “self-describing JSON” which tells the Iglu where to locate the JSON schema that has to be used for validation purposes. Since the name you used is not located on our Iglu Central, the configuration structure cannot be validated.

To answer your question directly, no, the name is predefined and is not custom unless you host the corresponding JSON schema in your own Iglu server (that is it could be located somewhere). The properties vendor and name are used to locate the JSON schema that describes the actual data - here, it is the configuration itself.

You can read more about “self-describing” JSON schema here.

5

I understand self-describing json schema; my misunderstanding was that I thought the $schema field itself was enough for snowplow to find the type of enrichment. Since the “self” field of other self-describing json schema has “name” and “vendor” fields meant for the name of the schema and the vendor creating it (e.g: the name of the company), it is confusing that in this case, “name” and “vendor” must always be that of the custom enrichment config and com.snowplowanalytics.snowplow.enrichments

Since that is the case, it would be helpful if that were specifically pointed out in the documentation: Custom API Request enrichment | Snowplow Documentation

and it would also be helpful if these fields were hardcoded to accept only those values during validation, using the “pattern” field of jsonschema e.g:

"name":
{
    "type": "string",
    "pattern": "api_request_enrichment_config"
}

etc. This would let the user know at load-time that their enrichment is invalid, instead of silently failing as it does right now

6

Hi @ShortRoundDev , I totally agree with you and will pass your suggestions to the relevant teams. Sorry for confusion caused.

8

It’s no problem; thank you for clearing things up for me!