Aug 11, 2008

MailChimp’s A/B Email Tester Can Be Really Mean

A lot of people tell us they appreciate the compliments that MailChimp gives them when they do their email marketing. Our chimp chatter feeds and little love notes seem to put a smile on our customers’ faces.

But this weekend, I discovered there’s a mean side of MailChimp, too…

I ran an A/B test on a recent campaign of mine, and I actually thought my Group B subject line would win (I thought more people would open a subject line that hinted at privacy policy changes, rather than new features). The A/B split proved my theory wrong. My results:
Group A Subject Line (31% open rate): MailChimp v3.2: RSS-to-email, Batch Unsubscribe, 16.5 languages…
Group B Subject Line (27% open rate): MailChimp v3.2: Privacy policy changes,  RSS-to-email, Batch Unsubscribe, 16.5 languages…

So I guess people aren’t as paranoid about their privacy as I am.  Good thing I did an A/B test, because the open rate for my crappy subject line would have meant almost 2,000 people wouldn’t have opened! If you’re an eRetailer, 2,000 lost opens can mean a lot of money.

Anyway, just to pour salt on my wound, MailChimp added a little commentary to my Group B attempt:

ab-split-loser.jpg

Aaah, nothing like being called a loser first thing in the morning. This is gonna be a good day.

Speaking of A/B testing, I hear that FutureNow’s book on A/B testing, "Always Be Testing" is now shipping.