I am looking to ingest Auth0 events into a GCP-based Snowplow installation. Auth0 is a popular identity management platform and supports near-real-time exports of user creation, login, … events using their log streams feature.
As Auth0 log events are not currently “natively” supported by Snowplow, I was considering using Snowplow’s remote adapter feature, but apparently remote adapters do not work with Beam Enrich, and their usage is somewhat controversial.
I see several alternatives, but would like some feedback on how to best proceed .
- Option 1. Implement an adapter in Scala; contribute this adapter to the Snowplow project.
Option 2. Send events to a custom endpoint/microservice of my own, which converts them into a “native”
snowplow_dataevent and then forwards them on to the default Snowplow
tp2-tracking endpoint. (Forwarding to
tp2seems to trump forwarding to the Iglu-adapter, mostly because it looks like the Iglu-adapter does not support batching of events; Auth0 micro-batches its events.)
I’m quite time-constrained, and as always having the feature faster is better. I have limited Scala experience. Implementing a small adapter micro-service only takes a matter of hours. All strong points in favor of remote adapters and now option two.
However, Auth0 is a popular, publicly available service. I’d like to contribute, and I’d like “official” integration. Looking at existing adapters, the actual programming required to ingest log events would also be limited; I’d probably spend more time setting up my dev env than coding. I’d go for this if there’s a good chance of the adapter ending up in some future release of common-enrich. I don’t want to deploy from source every time there’s a new release…
- Would an Auth0 event log adapter be a good candidate adapter? We’ve determined that we want to use Snowplow to ingest these logs, but log events are maybe not as structured as typical Snowplow data. There’s structure though - besides the log message itself, entries do have properties such as event type, user id, … It’s not just text.
- I’m not affiliated with Auth0; is “third-party” adapter development like this encouraged? Is it ok if I’m the one defining event types under the
com.auth0-namespace in Iglu central? I’m thinking of involving Auth0 or asking them to contribute, as integration with other services is their strong point. They integrate with Datadog, EventBridge, Event Grid and Slack; Snowplow should be there .
- What about option 2? It seems quite simple to me. It’s a fairly generic alternative to using the remote adapters, and actually seems easier to set up as it does not require any Snowplow configuration changes.