Jan 22, 2010

A/B Testing Email Ads

I’m pruning some bookmarks from my browser this morning, and came across this article on Clickz:

Tips for Selling Ads in E-mail by Jeanne Jennings.

Very useful stuff, if your master plan is to become the next FlavorPill, Thrillist or DailyCandy (purchased for $125M).

One of Jeanne’s tips is to allow your advertisers to run A/B tests within your campaigns. You probably know we’ve got an automated A/B testing tool, but did you know that if you use it in conjunction with our dynamic content merge tags, you can actually A/B test your content and creative (even your entire email design, if you wish)?

Let’s say one of your advertisers sells cardboard boxes (exciting!), and they’ve got this cute, kitten-tastic banner ad to run in your newsletter:

box-sale-cats

maybe it’s been running for quite some time, and giving them awesome ROI (see how to track ROI w/MailChimp + Google Analytics).

But now they’re wondering if puppies would perform better:

box-sale-dog

Here’s how you can help them run an A/B test on their creative.

Setup an A/B campaign in MailChimp like you normally do. I usually do a test on subject lines.

When you get to the campaign builder, open the text editor and switch to "Source" view. We’re going to be coding some merge tags:

source-view

Wherever you want to place your banner ad, use the block of example code from this blog post:

A/B test your design and content in MailChimp

It looks like this:

ab-code

When the sample code is in place, go back and replace a few things:

  1. Wherever you see "this block of content/graphics…" replace that with your banner ad code. Perhaps they have their image hosted somewhere (ad-server?) or you might be hosting that for them in your MailChimp image gallery (it’s free, and served through our CDN). Stick banner ad "A" in the "GROUP:A" slot, and the same for B.
  2. This sample code has spaces and hard returns to make it easy for you to read and understand. But you don’t want those spaces and hard returns to mess up your email formatting whenever it actually renders in the inbox. So ater your banner ad code is placed, go and delete all the extra space so that this is just one continuous line of code. Like this:
ab-code-sans-breaks

When you hit "save" and close the editor, you’ll see both of the banner ads in the preview pane. Don’t worry about that. It’s because the dynamic content engine doesn’t run until you actually send your emails.

So go ahead and send two tests to yourself, so you can see this in action:

send-ab-test

Notice there are separate buttons for group A vs. group B. Send tests to yourself using both, and you can compare them after they get to your inbox. Here’s what they look like side-by-side in my gmail:

compare-ab

If everything looks okay, go ahead and send 2 tests to your advertiser too, and show them how awesome you are.

Be sure to tell them you used this handy Subject Line Suggester to get the best open rate possible for them:

and that you used our Inbox Inspector tool to make sure there’d be no deliverability problems. If your list is big enough, you may want to opt for a dedicated IP address (here’s a nice case study on how that can help).

When the campaign has run its course, and it’s time to send them some reports, use our reports sharing feature (you can even re-brand it with your logo).

In particular, you’ll want to point out if your newsletter performed better than industry average.

We’ll show you an industry comparison right in your campaign report:

If you’re not a MailChimp customer, you can get this data free at: mailchimp.com/charts

More MailChimp features your advertisers will appreciate: