Support for POST payload for Iglu Webhook Adapter

For engineers Enrichment
1

Hi folks,

I was wondering the the Snowplow webhooks collector endpoint supported POST requests in addition to GET requests. The documentation and code suggest that POST requests may not work.

A quick test with an example schema yields the error “Querystring is empty: no Iglu-compatible event to process”, further suggestive that it’s only the query string being processed.

As there are quite a few services which only support POSTing payloads to a webhook URL what’s my best course of action here? I can think of three methods which might (?) be suitable:

  1. Write a new adapter for a com.snowplowanalytics.iglu/v2 endpoint which supports both GET arguments and POST.
  2. Write a new vendor adapter (similar to the existing vendor adapters now) specific to each service that are capable of extracting the body out of CollectorPayload.
  3. Writing an intermediate proxy service such that webhooks call a URL with the POST payload which is converted to a GET payload before being forwarded to the Iglu webhooks endpoint.

Method 3 is possibly the easiest method but not the cleanest because of the limitations of potentially converting a large/verbose POST payload into query parameters and although 1 and 2 seem more suitable I’m not a Scala guru (but the existing adapters seem to provide a good starting point).

Thoughts?

1 Like
2

Hey Mike,

It’s a great question - for other readers, to clarify Mike is talking about Snowplow’s Iglu Webhook Adapter, which lets you send in any data on a GET querystring as long as it includes a &schema= parameter which defines the Iglu schema URI.

The simple answer is that we implemented the Iglu Webhook Adapter as GET initially because we were building it for a provider that used GET (Adjust). There’s no reason why the existing Iglu Webhook Adapter couldn’t be extended to support POST as well.

The nice thing about POST support is that you could support a straightforward self-describing JSON as the payload - which is cleaner than the GET support (which has to assume each querystring parameter is a string in the absence of any other type information).

So my suggestion is to go with option 1, but just extend the existing adapter, no need to write another one. And it would be fine to support just POST of a self-describing JSON (i.e. no need to implement application/x-www-form-urlencoded or multipart/form-data yet).

Does that answer your question?

3

Thanks Alex, yes that answers my question. I’ll attempt to look at handling POST payloads within the existing Iglu adapter.

4

Hi @mike @alex I see the Webhook still only supports GET, are there any developments on this?

5

@joaocorreia, to my knowledge Iglu Webhook has been supporting POST for very long time, Iglu Webhook | Snowplow Documentation

1 Like
6

Yeah @ihor is correct, I think I added POST support back in 2016 after this request.

2 Likes
7

Apologies, I meant the Iglu that fires the webhook when a schema is published @mike @ihor, the one you configure on the Iglu configuration file.

This PR implements POST support for the webhook: The config allows setting webhooks to use POST method by AlexITC · Pull Request #111 · snowplow/iglu-server · GitHub

# Optional webhooks configuration
  "webhooks": {
    # Enable webhooks that will be called when a schema is published or updated
    # with a vendor that matches the specified prefixes
    # with a vendor that matches the specified prefixes, allowing to use the
    # http GET (by default) or POST method.
    "schemaPublished": [
      # {
      #   uri = "https://example.com/endpoint"
      #   vendorPrefixes = ["com", "org.acme", "org.snowplow"]
      # }
      # {
      #   uri = "https://example.com/endpoint",
      #   usePost = true
      # },
    ]
  }