Today we’re publicly launching Sixpack, a language-agnostic A/B testing framework with an easy to use API and built-in dashboard.

Sixpack has two main components: the sixpack server, which collects experiment data and makes decisions about which alternatives to show to which users, and sixpack-web, the web-based dashboard. Within sixpack-web you can update experiment descriptions, set variations as winners, archive experiments, and view graphs of an experiment’s success across multiple KPIs.

Why did we do this?

We try to A/B test as much as possible, and have found that the key to running frequent, high-quality tests is to make it trivial to setup the scaffolding for a new test. After some discussion about how to make the creation of tests as simple as possible, we settled on the idea of porting Andrew Nesbitt’s fantastic Ruby Gem ‘Split’ to PHP, as the templating layer of the SeatGeek application is written in PHP. This worked for a bit, but we soon realized that only being able to start and finish tests in PHP was a big limitation.

SeatGeek is a service-oriented web application with PHP only in the templating/routing layer. We’ve also got a variety of Python and Ruby services, and plenty of complex JavaScript in the browser. In addition have a WordPress blog that doesn’t play nicely with Symfony (our PHP MVC) sessions and cookies. A/B testing across these platforms with our PHP port of Split was a hassle that involved manually passing around user tokens and alternative names.

If, for example, we wanted to figure out which variation of content in a modal window on our blog (implemented in JavaScript) led to the highest rate of clicks on tickets in our app (implemented in PHP), we’d need to create a one-off ajax endpoint to register participation and pass along a user token of some sort into the Symfony world. This kind of complexity was stopping us from running frequent, high-quality tests; they just took to long to set up.

Ideally we wanted to be able to start a test with a single line of JavaScript, and then to finish it with a single line of PHP. Since there was no tool that enabled us to do this, we wrote Sixpack.

How does is work?

Once you install the service, make a request to participate in an experiment like so:

$ curl http://localhost:5000/participate?experiment=bold-header&alternatives=yes&alternatives=no&client_id=867905675c2e8d54b6497ea5635ea94dca9fb415

You’ll get a response like this:

{
    status: "ok",
    alternative: {
        name: "no"
    },
    experiment: {
        version: 0,
        name: "bold-header"
    },
    client_id: "867905675c2e8d54b6497ea5635ea94dca9fb415"
}

The alternative is first chosen by random, but subsequent requests choose the alternative based on the client_id query parameter. The client library is responsible for generating and storing unique client ids. All of the official SeatGeek client libraries use some version of UUID. Client ids can be stored in MySQL, Redis, cookies, sessions or anything you prefer and are unique to each user.

Converting a user is just as simple. The request looks like:

$ curl http://localhost:5000/convert?experiment=bold-header&client_id=867905675c2e8d54b6497ea5635ea94dca9fb415&kpi=goal-1

You don’t need to pass along the alternative that converted, as this is handled by the sixpack server. The relevant response looks like this:

{
    status: "ok",
    alternative: {
        name: "no"
    },
    experiment: {
        version: 0,
        name: "bold-header"
    },
    conversion: {
        value: null
        kpi: "goal-1"
    },
    client_id: "867905675c2e8d54b6497ea5635ea94dca9fb415"
}

As a company we aren’t only interested in absolute conversions; we’re interested in revenue too. Thus, the next Sixpack release will allow you to pass a revenue value with each conversion which sixpack-web will use to determine a revenue-optimized winner of the experiment.

Clients

We’ve written clients for Sixpack in PHP, Ruby, Python and JavaScript which make it easy to integrate your application with Sixpack. Here’s an example using our Ruby client:

require 'sixpack'
session = Sixpack::Session.new

# Participate in a test (creates the test if necessary)
session.participate("new-test", ["alternative-1", "alternative-2"])
set_cookie_in_your_web_framework("sixpack-id", session.client_id)

# Convert
session.convert("new-test")

Note that while we must wait for a response from the participate endpoint to get our alternative necessary to render the page, we do not have to wait for the conversion action. By backgrounding the call to convert we can save a blocking web request.

What did we use to build this thing?

Sixpack is built with Python, Redis, and Lua.

The core

At the heart of Sixpack is a set of libraries that are shared between sixpack-server and sixpack-web. To keep things fast and efficient, Sixpack uses Redis as its only datastore. Redis’s built-in Lua scripting also gives us the ability to do some pretty cool things. For example, we borrowed ‘monotonic_zadd’ from Crashlytics for generating internal sequential user ids from UUIDs provided from the client libraries.

The Sixpack Server

We wanted to keep the server as lightweight as possible since making additional web requests for each experiment on each page load could quickly become expensive. We had originally thought to write Sixpack as a pure WSGI application, but decided that the benefits of using Werkzeug outweighed the cost of an additional dependency. In addition, Werkzeug plays very nicely with gunicorn, which we had already planned to use with Sixpack in our production environment.

Sixpack-web

Sixpack-web is slightly heavier, and uses Flask because of its ease of use and templating. The UI is built with Twitter Bootstrap, and the charts are drawn with d3.js.

How to get it and contribute

You can check out Sixpack here.

We’ve been using Sixpack internally at SeatGeek for over six months with great success. But Sixpack is young, and as such is still under active development. If you notice any bugs, please open a GitHub issue (http://github.com/seatgeek/sixpack), or fork the repo and make a pull request.