Since its release in late 2007, MailChimp’s A/B testing tool has been a huge success with our customers. In fact, over 200,000 split tests were sent in 2013 alone. Using subscribers as lab rats is still both fun and educational, but we realize that some of you might not know how to get started with the tool. To help you out, we put together a quick refresher on getting the most out of A/B testing. We’ll also share some of our research on the best ways to test.

What is an A/B test?

A/B testing allows you to send two different versions of your campaign to part of your list, and then select the version that gets the most opens or clicks to send to the rest of your list. MailChimp can run the tests automatically for you, so all you have to do is tell us what to test and what percentage of your list you want to test it on. By changing aspects of the campaign between the two different groups, you can find out what your subscribers respond to. What can you test this way?

• Subject line – Try different wordings to see what gets the most attention
• From name – See if your subscribers prefer mail from a person or an organization
• Delivery date/time – Figure out when your subscribers like to open/click
• Content – Test different images, layouts, and messages

We recommend testing only one difference between the A and B groups. When there are several differences between test groups, it’s difficult to figure out which change impacted your engagement. If you make only one change, you’ll know exactly what caused the difference.

What kind of results can I expect?

We examined A/B tests sent in the past year to measure their impact. For subject lines and from names, we calculated a Levenshtein Distance to measure the difference between test values. For send time, we looked at the absolute difference in hours. Percentages were calculated as a percent of opens or clicks, and not as a percent of emails sent, so a 10% increase on 20 clicks would bring you up to 22.

Content Tested	Tests Run in the Past Year	Avg. Open Improvement	Avg. Click Improvement	Avg. Difference
Subject	209,824	9.0%	11.0%	32 Character Changes
Send Time	25,861	9.3%	22.6%	10 Hours
From Name	6,203	12.0%	15.3%	12 Character Changes

The majority of tests were on subject lines, and the average Levenshtein distance of 32 suggests that most tests use very different subject lines. Although subject line is the most popular test, send time and from name tests are powerful too. Testing send time and from name resulted in 22% and 15% increases in clicks on average. If those results pique your interest, you might like our Send Time Optimization tool.

Getting started

So how do you build an A/B test? Let’s walk through the steps using a recent campaign from MailChimp customer Baron Fig.

 

 

When creating a new campaign, you’ll see A/B Split Campaign as an option. For free accounts, this feature will only be available after sending a few campaigns and tracking results.

After selecting A/B split campaign, you’ll tell MailChimp what you want to test. Campaign content is not listed here because it’s controlled using the A/B merge tags. You’ll also determine your test segment size, how to choose a winner (clicks/opens/manual), and how long to wait before choosing a winner.

We recommend a test segment of 20%-50%. Smaller lists benefit from higher percentages, which test on more subscribers. Larger test percentages have a lower overall impact, however, because the winning campaign will be sent to fewer people.

How do you choose between clicks and opens? If your campaign has clickable links, all of the items we mentioned can impact your click rate. Campaign content never impacts your open rate, though. We think testing on clicks makes the most sense for people who have links or want to test campaign content. Higher open rates also typically lead to higher click rates. It’s also worth noting that changing the number of links in your campaign can impact your click rate and might confound your results. If you want to have complete control over which campaign is chosen, you can tell MailChimp to wait until you manually select a winner.

	Subject	From Name	Send Time/Date	Campaign Content
Clicks
Opens

We also recommend waiting at least a couple of hours before choosing a winner. If you only wait one hour, you won’t give everybody in your test group time to open your email and click on links. The winner after one hour might not be the winner after several hours.

Providing your test values

Once you’ve set up your test and selected a list to send to, you’ll need to provide your test values. You can choose to test very slight differences, like punctuation, or very substantial differences, like completely different subject lines or campaign layouts. If you’re trying out some subject lines, you can even use the subject line researcher or review some of our latest research on what makes a good subject line. Baron Fig decided to test two very different subject lines for their campaign.

Once this is done, all you need to do is build the rest of your campaign, send it out, and wait for the results!

For Baron Fig’s campaign, group A won by a statistically significant margin. To interpret this result, you need to figure out what the difference between the campaigns boils down to. Group A received a bold announcement of new information (Meet the Confidant), which is something that they can act on now. Group B received an announcement of a future event that they can’t really act on yet (Store Opens in 7 Days). The subscribers may have been more compelled to click on subject A because they knew they would immediately get something out of it. With that big of a difference in performance, we’re certainly glad the rest of the list saw the subject line that group A did.

What happens if your open or click rates aren’t that different? It could mean your subscribers were indifferent to the change you made, but it never hurts to test again. Repeating a test is a great way to make sure the change has a consistent impact. Each time you test, you’ll get a new randomly selected test group to use as lab rats!

Go forth and test

Now that you know the basics of A/B testing, the only thing left to do is dive in. As you continue to test, you’ll start to get a good grasp of what your subscribers respond to, and what doesn’t work for them. This might change as your list grows, though, so it is always a good idea to keep testing.