Dec 22, 2007

A/B Testing Emails Catching On

When we launched our new patent-pending A/B email testing feature, most of our customers would respond with, "What’s A/B testing?" When we explained how A/B testing works, and how MailChimp makes it totally automatic (and free), they’d respond with, "Holy $#@%. Why would I not do A/B testing for every single campaign?"

After that, they test religiously.

It’s catching on. And after Dec.31st, A/B testing will be available to ALL our customers—even our advanced users who copy-paste their own HTML code. Here’s the upgrade announcement, in case you missed it (previously, it was only available if you used our built-in HTML email templates).

We just ran some stats on our system, and discovered that since Nov.7, 223 A/B test campaigns have been sent, to over 1.2 million recipients.

These are mostly small businesses who, without MailChimp, would have absolutely no budget (let alone time) to run sophisticated A/B email tests. These small businesses are learning what time of day gets the best results for their audience. What style subject lines work better. Whether or not they should personalize their "From:" fields. And all it takes is a click of this little button:

A/B Test button in MailChimp

What I’m really excited about is how we’re opening A/B testing up to all our advanced users. These are the web designers, freelancers, and creative agencies who send campaigns on behalf of all their clients. They’ve been itching to get hold of this functionality since we launched, and it’s going to be fun seeing what they test, and hearing what they’ve learned. And yes, we’ve been getting all your expert feedback, and yes, we are definitely (and constantly) going to be adding more testing features to it.
We’re compiling a bunch of stats from our customers, and will post some A/B test results soon. They’re fascinating (and it’s fun to guess the A vs. B winner). Stay tuned. Most of the subject line tests reflect what we’ve told people for years: tell what’s inside, don’t sell what’s inside. And in most cases, finding some way to include your company name (or some way for recipients to instantly recognize you) helps with open rates.