Alex Kehayias's Blog

A/B Split Testing with Django

Dear internet startups, there’s no excuse not to A/B test your shit. You will learn horrible things about yourself. For example, that “really cool feature that you added that is so awesome people are going to be glued to my app,” doesn’t actually do anything to improve your core metrics. In fact they may be hurting them. Since it’s hard to test apples to apples you are flying blind. 

Don’t worry, I’ve done that too. It’s dumb now that I look back and I should have started testing everything sooner. I thought it would be too hard to do, to hard to implement, and slow down my lightning development skills (ha!). It’s worth the up front investment because you can help correlate the things you do to lasting results.

A/B test with django-lean

I don’t use Google’s wonderful website optimizer. It doesn’t help me do feature a/b tests. If you want to test the copy on your homepage then fine use GWO, but if you want to test that sweet feature you just added good luck trying to do that with optimizer. I use django-lean on just about all my projects to do real A/B tests that use a chi squared analysis to measure for lasting improvement changes. It makes it very easy to make experiments out of chunks of functionality you are using and figure out if what you are doing is moving the needle. Oh and you can run multiple experiments at the same time (that’s not multivariate testing, that just means multiple unique experiments).

Modifications

I only use the “experiments” app that’s in django-lean and remove everything else. I haven’t needed the other stuff in there and it’s not documented.

Setup

I put my modified version of django-lean into my project folder rather than installing it as a dependency in site-packages. I’ve made changes to it and probably will make more so I’d rather treat it like a project app. Follow their wiki to add the urls to your url conf and syncdb/migrate. 

It was really tricky figuring out just how to enroll visitors/users into an experiment. Once I figured it out I made a wrapper for it that can be added to any view that you want people to be added to an experiment. For me that’s just about every view. Here’s my decorator that wraps views to enroll the user in an experiment if they are not already.

# Note that I modified the paths from django_lean.experiments to experiments

from experiments.utils import WebUser

def set_experiment_user(target):
    '''Decorator for setting the WebUser for use with ab split testing
    assumes the first argument is the request object'''
    def wrapper(*args, **kwargs):
        request = args[0]
        WebUser(request).confirm_human()
        return target(*args, **kwargs)

    return wrapper

Now we can use this in our views like so:

# My wrapper is in lib.utils, but put it wherever you want
from lib.utils import set_experiment_user

@set_experiment_user
def home(request):
    '''Your view here as per usual'''
    return direct_to_template(request, 'index.html', locals())

Now anyone that goes to home will be enrolled in whatever active experiments I have going on. 

We also need to add the following to our templates. A script that attempts to figure out if you are a real visitor and not a bot and boilerplate stuff for the experiments:

<script language="javascript" src="{% get_static_prefix %}javascripts/experiments.js" type="text/javascript">
{% include "experiments/include/experiment_enrollment.html" %}

Creating and experiment

Experiments have two paths, control and test. In your templates you use the experiments tags to show chunks of code that are for the test or for the control in the experiment. 

{% load experiments %}
{% experiment experiment_name control %}
This is shown when the visitor is in the control group
{% endexperiment %}
This is shown when the visitor is in the test group
{% experiment experiment_name test %}

In your views you can route to the right code like this:

    if Experiment.test("experiment_name", WebUser(request)):
        # do some stuff knowing they are in the test group

Recording goals

We need to record the key actions the user makes so that we can compare them in the A/B test. These should be aligned with the core measurements you need to make to know what your users are doing. Don’t track goals that make no difference in figuring out if you are better engaging your customers. Focus on actions, for TimeoutDebate that means votes, opinions, and number of debate views. You add goals through the admin and can call them whatever you like. To record that a user has performed the action you do this:

from experiments.models import GoalRecord, Experiment

# In your view somewhere
GoalRecord.record("vote create", WebUser(request))
# This records the "vote create" goal for that experiment participant

Also you can do this using a tracking pixel that points to your django-lean url route (not the admin one). Simply add an img tag with the source of a url that points to your goal and it will record it just like the above. This is useful for putting it inside ajax interactions or other places that are not controlled by your django views. 

Engagement Calculator

This is an interesting calculation that gets added to the A/B report that is generated by django-lean. You basically write a calculation that returns an arbitrary number based on actions a user has taken that measures overall engagement with your system. I don’t like the example on the django-lean wiki because it doesn’t reuse the goals recorded to calculate the user’s engagement score. This example reuses it and awards points accordingly:

# Defines the engagement score for experiments
from experiments.models import GoalRecord

class EngagementScoreCalculator(object):
    def calculate_user_engagement_score(self, anonymous_visitor, start_date, end_date):
        """
        Defines the a user's engagement based on the actions they take
        Points awarded for certain actions.

        """
        # Sets the weighting for the engagement score
        weighting = {
            "signup": 10,
            "debate create": 0,
            "debate view": 1,            
            "opinion create": 10,
            "vote create": 5,
        }
        # Get all the tracking goals completed by them over the time period
        goals_completed_set = GoalRecord.objects.filter(created__range=(start_date, end_date), anonymous_visitor=anonymous_visitor)
        # Get a list of all the GoalRecord Names
        tracking_goals = {}
        for i in GoalRecord.objects.all():
            tracking_goals[i.goal_type.name] = 0
        # Count all the goals
        for i in goals_completed_set:
            if i.goal_type.name in tracking_goals.keys():
                try:
                    tracking_goals[i.goal_type.name] = tracking_goals[i.goal_type.name] + 1 * weighting[i.goal_type.name] # multiply by the weighting
                except:
                    tracking_goals[i.goal_type.name] = 0
        days_in_period = (end_date - start_date).days + 1
        # Sum up the total
        total = 0
        for key in tracking_goals.keys():
            total = tracking_goals[key] + total
        engagement_score = (float(total)/days_in_period)
        return engagement_score

Generate reports and make decisions

Update your experiment reports using the django-lean management command “python manage.py update_experiment_reports” Now when you look in the admin (mine is /admin/django-lean) you will see a list of experiments. When you click on it you will see the report with all the lovely statistical analysis. It even makes a check mark next to items that have a confidence interval above 95% (that means that it is 95% confident that the test version will have a lasting improvement). 

To make decisions, make sure you have a large enough sample size (number of people enrolled in your experiment in both the test and control group) and look at the confidence intervals, improvement, and engagement score. If it’s not conclusive just keep measuring it. Green is generally good, red is generally bad when it comes to the experiment report. The key here is that you’re making the decision based on real data, not your gut. You can disable your experiment at any time in the admin and “promote” the winning version. Just make sure you clean up your templates and views so it doesn’t get super cluttered with experiments.