Sep 17, 2015

Multivariate Testing for MailChimp Pro

As MailChimp’s Chief Data Scientist, the main reason I love my job—on top of all the other many, many fantastic reasons to work here—is that sending email generates lots and lots of data. Opens, clicks, purchases—this is the kind of stuff that businesses can use to refine their marketing. Last year alone, recipients of emails sent through MailChimp clicked 3 billion URLs. That’s a lot of customer engagement!

And it just got even better. This week, we launched MailChimp Pro, a robust set of powerful features for users who want to be more sophisticated and intentional in using customer engagement data to better understand and serve their audience. MailChimp Pro is packed full of tools for users who love data as much as I do. (Spoiler alert: I love it a lot.) One of those features is Multivariate Testing, and I want to tell you all about it.

Way beyond A/B

Our A/B testing tool is popular. Like, Mean Girls popular. Last year, MailChimp sent 12 billion A/B test emails—which means we helped hundreds of thousands of businesses around the world perform marketing studies on billions of their customers. Now, MailChimp Pro users will have access to a pumped-up version of A/B called Multivariate Testing. With Multivariate, rather than testing 3 different campaigns, users can test up to 8 at a time.

With standard A/B testing, users can only test single design elements. But with Multivariate, users will be able to combine subject lines, from names, content, and send times simultaneously to see which approach works best for boosting open rates, click rates, or even top line revenue.

We go to 11 (well, 8)

Time for me to let you in on a secret: We’ve already been experimenting with Multivariate in our own emails. Last month when we announced the release of the A/B testing update for all customers, we secretly sent the announcement email as a Multivariate campaign with 8 different versions of the content. Sneaky!


You can test templates with Multivariate, but we decided to run this one as a pure content test. We experimented with length, headlines, and images. We made one version purple, because Ron, our Creative Director, hates purple. The purple campaign received thousands more clicks, earning double the click rate of the other campaigns. So we celebrated by painting everything purple, starting with Ron’s car.

Testing, testing

With MailChimp Pro, we’ve taken the idea of A/B testing and turned the knob to 8. But Multivariate Testing isn’t just about “8 instead of 3.” The “multi” is all about how you get up to 8 possible campaigns. While you can simply test up to 8 versions of the same thing (like we did with our content in the grid above), you can also test combinations of multiple email components: subject line, from name, send time, and content.

Rather than testing 8 different versions of content, I might want to test, for example, 2 subject lines and 2 different content options. That’s a total of 2 x 2 = 4 total versions of the campaign: subject A with content A, subject A with content B, subject B with content A, and subject B with content B.

Asthmatic Kitty Records recently beta-tested Multivariate and decided to combine 2 different subject lines with 4 different content options for a total of 8 different test campaigns. Here’s what a few of them looked like:



Campaigns 1 and 2 have the same subject line, as do campaigns 3 and 4. Campaigns 1 and 3 have the same content, as do campaigns 2 and 4. This allows them to test subject lines and content all in the same mailing.

Better reports, better insights

After you run a test, it’s always fun to see which version won. But when you’re combining elements into a Multivariate campaign, it’s also important to tease out which element was most important in the winning campaign. Was it the subject line change that mattered? Or the content change? We’ve set up Multivariate reporting to provide these answers.

There are 2 sections in each Multivariate report: Combination results and Results per variable. In Combination results, we show the stats of each test version of a Multivariate campaign so you can see which combination of elements won.

In the Results per variable section, individual variations are broken out so that you can see how much the change mattered across all versions of the campaign it went into. In our Asthmatic Kitty example below, we see that the “Want to see Sufjan live?” subject line did far better than the other subject line across all the test campaigns it was placed in. The 0.7% increase in click rate for that subject line stands out versus the 0.3% click rate increase for the winning content variation over the next highest performing.




In Multivariate, we’ve also added confidence intervals (think of these like the margins of error that are reported with presidential poll numbers) to each statistic for those who want to separate statistical ties from sure things.

It’s somewhat about the Benjamins

With the release of Multivariate, we’ve added a new metric that users can test on: money! It’s helpful to know which marketing content not only drives engagement but actually drives purchases. And for those Pro users who integrate their e-commerce data with MailChimp, that’s now possible.

Purchases on an online seller’s site can be tracked all the way back to the Multivariate test campaign a specific email address received. Then, a winning campaign can be selected based on revenue, and that campaign can be sent to the rest of the list.

Using data science to ensure success

This is my favorite section of this blog post, but real talk: it’s pretty dry. TL;DR: We studied billions of A/B test emails to create sample size and wait time recommendations for you. If that excites you as much as it excites me, read on. I’ll catch up with the rest of you in the section below.

So, we recommend that you send a test to your whole list with the objective of learning generalizable strategies that you can use over and over again. For instance, “my audience responds well to large single-image campaigns” is something you can use over and over again. On the other hand, “my audience loves purple now and always” may not stand the test of time.

That said, many of our users aren’t interested in learning so much as they’re interested in improving the performance of each send via testing elements specific to that campaign only. They want to test against a piece of their list, declare a test winner, and send the winning campaign to the rest of their list. And with the ability to test 8 versions of content, content optimization through testing has never been easier.

But when you conduct a test where you act on the winner, you need to make sure that the selected winner at the end of your test maintains its dominance once all the opens and clicks have rolled in. Sometimes users set test durations too low and sample sizes too low to get accurate engagement data for declaring a test winner. So we studied our historical data (historical engagement rates, sample sizes, test durations) and came up with some recommendations for Multivariate tests.

When running a test, we recommend at least a 4-hour duration before declaring a winner. That’s to ensure enough opens and clicks roll in. Millions of historical test campaigns’ data tell us that you are twice as likely to select the wrong winner at 1 hour post-send as you are at 4 hours, so keep that in mind.

We also recommend sample sizes of 5,000 or more subscribers when conducting Multivariate tests, but this value will vary based on the typical engagement of your list, the metric you’re testing, and other factors. We recommend using a sample size calculator if you wanna truly get jiggy wit it.

Testing, learning, and acting

Our intention in building this tool—and all the other tools that make up MailChimp Prois to make rich data available to our users, so they can know more about their customers and use that information to grow their businesses. We know everyone will use the feature in different ways, some in an ad hoc manner and some as a consistent practice. I’m a biased data nerd, of course, but I recommend including test elements in every campaign, employing Multivariate Testing as part of an “always be learning” approach to marketing.

For most businesses, their audience is constantly changing and evolving. Old customers disengage, new customers sign up for the newsletter, new products and merchandise capture the imagination—it’s the circle of life. By engaging in ongoing testing, Multivariate allows a business to tap into that evolution and listen in on customers’ changing tastes and preferences.

Well, that’s enough from me. It’s your turn, now—go send some Multivariate campaigns! (That way I can analyze them and write a follow-up to this post. Job security, y’all.)