How to conduct A/B testing in Snowplow

For data modelers & consumers
1

Hi everyone,
I have deployed snowplow and it is working greatly.
At the same time I went through many post and articles and want to learn how to run A/B testing with Snowplow. For example, how can I separate the traffic into Control and Test user group. Please let me know if need to put code into frontend or we need another tool to help Snowplow. Thank you

2

Hi Team,
I found this open source tool, mojito:

Not sure if this is something that you are using? Or even a better tool. Thank you

2 Likes
3

You can of course run custom testing but I know @robkingston has put a lot of effort into Mojito and integrating in with Snowplow so I would say that’s a really good starting point.

2 Likes
4

Thanks @mike and @kuangmichael07 - Mojito is indeed our open source toolset for running tests in Snowplow. Been using it since ~2013 on client sites.

When you say you’re looking for an A/B testing tool to run with Snowplow, are you just after traffic splitting?

Technically any testing tool will run with Snowplow, as long as you track the “assignment” (control/treatment) - e.g. see our JSON Schema here. If you have a testing tool of choice, you could easily hook any of them up to Snowplow in minutes.

But if you’re looking for a front-end splitting tool, I may be biased, but I think Mojito’s library is worth a look because:

  • Just 5.5kb for the base library gzipped (vs Google Optimize @ 25kb)
  • Built-in support for error tracking & error handling - another great use of Snowplow
  • All your experiments’ logic and code are source controlled & stored in git for easy development, code review and deployment
  • Your experiments are hosted by you (perfect for PCI compliance… and for avoiding being held hostage to pricing changes by 3rd parties)
4 Likes
5

Hi @robkingston,
Thank you so much for the answer.
One difficulties that I am running in is that we have 3 different platforms, web, ios and android. I looked into Mojito but I still need some support for mobiles. do you have any suggestions? Thank you!

6

You’re welcome @kuangmichael07.

Tough question! We’re not so familiar with app/server-side unfortunately (it’s something we’ve thought about but it’s not something we’ve needed to build at Mint Metrics).

You’ve basically got two options:

Option 1 - Get something off the shelf

Get Optimizely etc. across the apps and hook it up to Snowplow using self-describing events so you can report against all of your event data (not just what’s in the testing vendors’ limited analytics tool).

Option 2 - Roll your own decisioning

As a Snowplow user though, you’re lucky enough to have your tracking SDKs & measurement sorted across apps and web. You just need the decisioning logic - E.g.:

For this {{user ID}}, give me the decisions for 1) inclusion & 2) assignment on this {{experiment ID}}:

  1. Inclusion - Accept the user into the test
  2. Assignment - Put the user into a bucket - control/treatment

At assignment you’d fire your exposure/assignment event to identify when the user enters the test. For this, you can fire your self-describing experiment events here.

We’ve written about decisioning a bit. Mojito allow users to customise their own decisioning logic:

We don’t have the immediate incentive to build this out - but I wish we did.

7

This is a great answer @robkingston. We will do a deep dive into this area and hopefully we might be doing some open source projects in the future. Thank you again

1 Like
8

for server side tracking you could have a look at this framework - PlanOut | A framework for online field experiments - depending on which lang you use there are more implementations

1 Like
9

Yes, good point @evaldas - I forgot about PlanOut. Their Python experiments API is quite nice.

And since PlanOut is open source, it’s interesting to see how they implement the decisioning / hashing here:

2 Likes
10

@robkingston how would you say that PlanOut compares to Mojito for JavaScript?

I understand that Mojito is built on top of Snowplow (WIN) but how do they compare functionally?

1 Like
11

They’re quite different and you can connect them both to Snowplow through self-describing events. We haven’t used PlanOut for any projects here, so maybe @evaldas can chime in about PlanOut’s JS lib if I get something wrong here.

I think the key difference lies in test code development:

PlanOut JS

  • Experiment code is inside your frontend code
    • Build experiments into your frontend
    • Start/Stop/Archive experiments when you deploy your site
  • When you find a winner you would need to remove the experiment logic but hardcoding is basically done
  • Focussed squarely on splitting and tracking - no extra utilities for transforming the DOM
  • Use PlanOut’s decisioning - not customisable and you may want to use their compatible library so the decisioning works cross-platform

Mojito JS

  • Experiment code is managed separately from your frontend code
    • Build experiments over the top of your frontend
    • Start/Stop/Archive experiments independently of site deployments
  • When you find a winner, you need to hardcode it into the frontend
  • Includes utilities for transforming the frontend and tracking/protecting users from errors
  • Use Mojito’s decisioning or implement your own custom decisioning (e.g. for partitioned ramping)

Both tools are just feature toggles with RNG.

Regardless of PlanOut / Mojito / Optimizely, the big win is tracking experiments into Snowplow. There you can analyse experiments against all your data (not just that data the testing tool has).

1 Like
12

@robkingston thanks for a summary of both frameworks. Honestly I actually never used their js implementation but I would guess it should work similar to the server side one. We currently use only on backend java implementation with few tweaks - GitHub - stacktome/planout4j: Java port of Facebook's PlanOut A/B testing system with additional functionality - as its not supported anymore, but honestly there is not much more to add to experiment framework as mostly you just want it to handle variations picking correctly which planout seems to do more or less as expected (at least java version).

Regardless of PlanOut / Mojito / Optimizely, the big win is tracking experiments into Snowplow. There you can analyse experiments against all your data (not just that data the testing tool has).

  • I completely agree on this point, I think the main thing that any framework should have ability to track any part website front-end/backend and being able to join it to the rest of the data is definitely there most value comes from compared to any other out of the box experiment solution.
1 Like
13

Thanks @robkingston and @evaldas for your detailed explanations - super-interesting!