Aug 26, 2008

A/B Split Testing – Does it Help Email Marketing?

Ever since we launched our automatic email optimizer, we’ve been tracking all this wonderful A/B testing data on our system.

We analyzed 1,720 A/B tested campaigns, sent to 6,226,331 inboxes (we only included campaigns sent to over 500 recipients).

Overall, we found that A/B split winners did better than non-tested campaigns. By how much?

Before we give you the exact numbers, test your email marketing aptitude and pick the winning subject line from these real life A/B email tests conducted by our customers:

Winning Subject Lines by Open Rate
Test Group "A" Test Group "B"
Don’t Forget Mom… Gifts Mom Will Love!
Protect Yourself From the Sun This Summer… The Best Sunscreen, Hats, Rash Guards and More…
[COMPANY]: March [COMPANY] Mail Spring Update from [COMPANY]
[COMPANY] March Newsletter March is National Crochet Month
[COMPANY]: France River Cruise & Golf – Last Call [COMPANY]: Last Call for France River Cruise & Golf
Computer Security Quarterly – Spring ’08 2007 Record Year For Data Breaches
Coupon Enclosed – Save on Printer Ink, Toner, Paper, Storage Media Now – HP, Xerox, Canon, Brother, Dell & More News & Discounts from [COMPANY] – Savings on Printer Supplies from HP, Xerox, Brother, Canon and More
[COMPANY] Easter Newsletter – Free Shipping [COMPANY] Easter Newsletter

Think you know which ones were the winners? They’re all the subject lines in the first (A) column. In general, we found that:

  • Subject lines with company names in them did better, and they did better when the company name was near the beginning of the subject line (perhaps it was the increased recognition factor)
  • Shorter subject lines seemed to work better than long subject lines (the difference in open rate was more noticeable when they differed by 30 characters)

How Much Better Are A/B Tested Email Campaigns?

We compared the email marketing stats of A/B testers against the average email marketing stats across all MailChimp customers.

Average open rate across all lists: 21.7%
Average open rate for winning A/B groups: 24.1%

Average click rate across all the lists: 4.7%
Average click rate for winning A/B groups: 5.5%

*We analyzed 1,720 A/B tested campaigns, sent to 6,226,331 inboxes (we filtered down to emails sent to over 500 recipients).

As you can see, those who take the time to A/B split test their email campaigns get better overall results than those who don’t. For some extremely large lists, those few percentage points can make a huge difference in eyeballs, clicks and conversions.