Ever since we launched our automatic email optimizer, we’ve been tracking all this wonderful A/B testing data on our system.
We analyzed 1,720 A/B tested campaigns, sent to 6,226,331 inboxes (we only included campaigns sent to over 500 recipients).
Overall, we found that A/B split winners did better than non-tested campaigns. By how much?
Before we give you the exact numbers, test your email marketing aptitude and pick the winning subject line from these real life A/B email tests conducted by our customers:
| Winning Subject Lines by Open Rate | |
| Test Group “A” | Test Group “B” |
| Don’t Forget Mom… | Gifts Mom Will Love! |
| Protect Yourself From the Sun This Summer… | The Best Sunscreen, Hats, Rash Guards and More… |
| [COMPANY]: March [COMPANY] Mail | Spring Update from [COMPANY] |
| [COMPANY] March Newsletter | March is National Crochet Month |
| [COMPANY]: France River Cruise & Golf – Last Call | [COMPANY]: Last Call for France River Cruise & Golf |
| Computer Security Quarterly – Spring ’08 | 2007 Record Year For Data Breaches |
| Coupon Enclosed – Save on Printer Ink, Toner, Paper, Storage Media Now – HP, Xerox, Canon, Brother, Dell & More | News & Discounts from [COMPANY] – Savings on Printer Supplies from HP, Xerox, Brother, Canon and More |
| [COMPANY] Easter Newsletter – Free Shipping | [COMPANY] Easter Newsletter |
Think you know which ones were the winners? They’re all the subject lines in the first (A) column. In general, we found that:
- Subject lines with company names in them did better, and they did better when the company name was near the beginning of the subject line (perhaps it was the increased recognition factor)
- Shorter subject lines seemed to work better than long subject lines (the difference in open rate was more noticeable when they differed by 30 characters)
How Much Better Are A/B Tested Email Campaigns?
We compared the email marketing stats of A/B testers against the average email marketing stats across all MailChimp customers.
Average open rate across all lists: 21.7%
Average open rate for winning A/B groups: 24.1%
Average click rate across all the lists: 4.7%
Average click rate for winning A/B groups: 5.5%
*We analyzed 1,720 A/B tested campaigns, sent to 6,226,331 inboxes (we filtered down to emails sent to over 500 recipients).
As you can see, those who take the time to A/B split test their email campaigns get better overall results than those who don’t. For some extremely large lists, those few percentage points can make a huge difference in eyeballs, clicks and conversions.
I’d be interested to know what the results were for the “losing” versions as well – ie how much lower than the average were the losers?
I’m assuming for this A/B testing, a smaller segment of the list is being used for the split test, then the winner being sent to the balance of the list? (As opposed to split-testing the whole list?)
P.
[...] and it’s unbelievably easy and automatic. Even a little fun. And people who use it get better results than people who don’t. It’s called A/B testing, and here’s how it works. Spread [...]
[...] the mantra of subject line length. MailChimp, for example, reviewed a host of campaign stats and found [...]
[...] recently, MailChimp evaluated the results of numerous A/B tests and [...]
The series B looks like better sales copy. Split testing email is a nice feature but a well written sales letter will always perform.
It would be interesting to see more information regarding subject lines. That is where the rubber meets the road….
@robertmstanley
I love you guys offer this in very easy format. thanks!
Wonder why “delivery time” option is 50/50 split and we don’t have an option to test smaller segment, then the rest goes out to the winner?
The problem is for non-american/engish users. This doesn’t work for the subject lines in spanish or other language.
It’s awesome how a simple subject line can change so much if you write it a way or another. Interesting post, I’ll try A/B split testing, we’ll see how it helps. Thank you from Argentina.
Excellent article
thanks for the tips