We analyzed 1,720 A/B tested campaigns, sent to 6,226,331 inboxes (we only included campaigns sent to over 500 recipients).

Overall, we found that A/B split winners did better than non-tested campaigns. By how much?

Before we give you the exact numbers, test your email marketing aptitude and pick the winning subject line from these real life A/B email tests conducted by our customers:

Think you know which ones were the winners? They’re all the subject lines in the first (A) column. In general, we found that:

• Subject lines with company names in them did better, and they did better when the company name was near the beginning of the subject line (perhaps it was the increased recognition factor)
• Shorter subject lines seemed to work better than long subject lines (the difference in open rate was more noticeable when they differed by 30 characters)

How Much Better Are A/B Tested Email Campaigns?

We compared the email marketing stats of A/B testers against the average email marketing stats across all MailChimp customers.

Average open rate across all lists: 21.7%
Average open rate for winning A/B groups: 24.1%

Average click rate across all the lists: 4.7%
Average click rate for winning A/B groups: 5.5%

*We analyzed 1,720 A/B tested campaigns, sent to 6,226,331 inboxes (we filtered down to emails sent to over 500 recipients).

As you can see, those who take the time to A/B split test their email campaigns get better overall results than those who don’t. For some extremely large lists, those few percentage points can make a huge difference in eyeballs, clicks and conversions.

If you haven’t already, I highly recommend you sign up for Conversion Rate Squirrel’s FREE “Life-Changing SuperConverters Newsletter”. Mainly because it’s full of useful tips that might help you with your own website. Also because it’s a fun, well-written newsletter that might give you ideas for your own email newsletter.

I asked the Conversion Rate Squirrel if he could offer up some tips to MailChimp customers for A/B testing their emails…

Here’s what the squirrel told me…

1. My top tip for any kind of split testing, is to make sure you have enough data to get a statistically significant result. Many of the multivariate & a/b split testing packages for testing websites have these already built-in… but too many people jump to conclusions before gathering enough data, and end-up basing a whole marketing campaign on some invalid results (I think that’s Tip 24 of our squirrel article).
2. In terms of the email content… I’m sure you guys already have a great understanding of ‘what to test’ (subjects, formatting, p.s’s etc…). Our approach is to spend loads of time learning about your websites visitors/customers – to understand their needs, and then make sure what you send them is what they want The results from our ‘Survey With A Difference’ email campaign, has steered our whole business and marketing plan
3. My final tip, we like to make sure what we send is entertaining – and always ask for feedback from our readers to make sure we’re not irritating anyone

A couple things got my attention.

• Interesting how they conducted a customer survey that “steered their whole business and marketing plan.” If you want to put together a survey of your own, be sure to check out our friends over at SurveyMonkey.
• They test content of their emails, which is a no-brainer. But one thing I don’t think I would have thought to test would be my “p.s’s” at the end of my messages. Interesting thought. I don’t even think I’ve ever written a P.S. in my newsletters, but if Conversion Rate Experts are doing it, I’m gonna try them too.

So right now, I’m looking at A/B split data across tons of campaigns. This is a real gold mine for email marketing research, so I was hoping to discover some universal truths that debunk what the “experts” tell us. Like, “Wednesdays are always best, not Mondays, so in your face!” Turns out, the only thing I’ve really learned is that all the expert tips out there on email marketing (mine included) are worthless. The results are all over the board. At first I was frustrated by the lack of consistency in all this data. Then, I realized that this is exactly the whole point—there is no silver bullet. Everything depends on your company, your offer, your list, etc. The only way to tell what works is to test, test, test.

But I am looking across all these amazing stats, and noticing some interesting patterns. For example, personalizing your subject line (such as with “FNAME”) doesn’t seem to help open rates very much. In fact, it can actually hurt…

Here are two representative campaigns that A/B tested subject lines with FNAMEs in them (I’ve disguised company names with “Acme”):

Example 1:
Sample A: Acme Energy Challenge: December Newsletter (40% open rate)
Sample B: *|FNAME|*, Your December Challenge Newsletter (37% open rate)

Example 2:
Sample A: Acme newsletter: Limited Quantity Parts (56% open rate)
Sample B: *|FNAME|*: limited quantity parts at special savings (46% open rate)

I do realize this is just two measly examples, but they pretty much represent what I’m seeing across all the campaigns in my data set. One subject line, then the same thing again with “FNAME,” or “FNAME:” in front of it. In almost every single case, the subject line with FNAME included was beaten by the subject line w/out it.

So Does FNAME Hurt?

I don’t think “FNAME” makes people not open. I think the problem is that FNAME is a waste of space in your already-cramped subject line slot. It’s just a hunch.

In examples 1 & 2, the FNAME tag basically replaced the company name in the subject line. And as we all know, your company name can help make your subject line look a lot more reputable and relevant (see: Subject Line Comparison Study).
Consider this example, where the winning subject line actually did include the recipient’s FNAME:

Example 3:
Sample A: *|FNAME|*, Get a laugh out of taxes, fire pricey ink, and more! (7.85% open rate)
Sample B: Get a laugh out of taxes, fire pricey ink, and more!’ (6.84% open rate)

Notice the winning subject line didn’t really win by all that much (sample size for A and B was 43,000 recipients each). Also notice that unlike the two campaigns above, this campaign’s subject line wasn’t really hampered by the FNAME. It didn’t replace the company name. In fact, there was no company name at all in either subject line. Maybe that’s why the overall open rate was so low (see: Subject Line Comparison Study)?

IMHO, the real “hook” of this subject line was the “Get a laugh out of taxes…” Everybody wants to laugh, right? And jamming an FNAME in front of that doesn’t really push that hook out of view, or replace it altogether. Could even be the case that using FNAME does help boost open rates for “informal” subject lines. But I have no idea. I guess you just have to run your own A/B tests to find out.