Jan 5, 2010

Can an ESP really help your deliverability?

inbox-graph-thmContrary to popular belief, switching to an ESP (email service provider) like MailChimp, ConstantContact, etc., is not a silver bullet for deliverability.

Spam filters and email gateways look at two kinds of things: 1) your email message, and 2) your email infrastructure. In terms of your email message, ESPs can’t help it if you send spammy content, manage your lists like an idiot, or if a link in your email has a bad domain reputation. But as ReturnPath points out here, ESPs do have one important advantage: our infrastructure.

So if you’re managing your own email servers and delivery issues with ISPs, firewalls, throttling, feedback loops, and bouncebacks are becoming a supreme headache, you might be thinking about moving to an ESP.

This article is about how one email sender, Photojojo, decided to switch from their in-house solution to MailChimp. We’ll show you how we helped them make the transition, what we experienced, and how it impacted their deliverability.

I know you’re just looking for the delivery stats, so let’s just cut straight to the chase.

Here’s what happened to Photojojo’s deliverability after switching to MailChimp:


Keep in mind that this is being measured by using ReturnPath’s seed list across 13 weekly campaigns. A seed list is simply a bunch of email addresses setup at various ISPs (and ReturnPath’s seed list covers more than 130 domains including the top ISPs in the U.S., European, Canadian, Latin American and Asia Pacific markets.).

Anyway, as you can see, Photojojo’s inbox% eventually got pretty high (in the 99% zone). There were even a couple instances of 100% (again, this is based on 100% of emails on ReturnPath’s seedlist receiving the message, not some absolute math or anything).

You may also notice a few other things, so let me add my notes to the chart:


and I’ll explain those notes in a minute.

Setting Expectations for the Switch

But first, some backstory. Before they switched to MailChimp, we learned that Photojojo had been sending emails from their own in-house system for a few years. This is pretty impressive, considering their list is in the hundred of thousands. They were managing unsubscribes and bounces themselves, but not quite as automagically as an ESP can.  So we warned them that when they switch, things wouldn’t be all good.

We set the following expectations:

  • Initially, they’d probably see a dramatic drop in list size, since our bounce cleaner would clean out all those dead email addresses. My experience in these situations is that list sizes get slashed by as much as 50%. Yes, you read that correctly: 50%.
  • Second, there might be a rise in spam complaints, if all of a sudden, old subscribers who weren’t receiving emails from Photojojo (because of delivery problems) actually started getting them.

In fact, our phone conversations kinda went like this in the beginning:

Me: Things are going to get worse before they get better, Amit. I’m warning you.

Amit: Yes, we understand there may be a decrease in our list size when we switch over. It’s to be expected.

Me: No seriously, you’re gonna hate us. You’re going to scream at me for ‘ruining your business.’

Amit: We know our bounce cleaning wasn’t great in the past, and we totally understand the repercussions. We’re fine.

Me: Yeeeeeeeah, right. We’ll see.

Preparing for the Switch

In an ideal world, we’d have a couple months to prepare for the switch. The reality was that we were only about 2 weeks (4 campaigns) away. With so little time, here’s what we recommended to prep Photojojo for the switch:

1. Warn all subscribers of the switch

We recommended they tell all current subscribers that Photojojo was about to switch email marketing systems. Photojojo should ask subscribers to add the Photojojo email address to their address books to prevent delivery issues. So at T-minus 4 campaigns, their pre-header was changed to include this message:


they sent 3 campaigns with that header, and on the last one before switching to MailChimp, they changed it to a more urgent looking:


a similar "heads-up" message was placed at the top of their website as well. If I had half a brain, I would have taken screenshots to show you those. You’ll just have to take my word for it.

2. Measure deliverability now, to establish the baseline

We gave Photojojo our ReturnPath seed list so that we could measure the deliverability of their in-house solution before switching to MailChimp, and then again after. After all, we need to know if MailChimp is even helping anything, right? If you look at the graphs above, the first 4 campaigns that you see are before MailChimp, from their own servers. All subsequent campaigns are sent from MailChimp. You can see a gradual rise in their inbox % after switching. Whew!

3. Warmup their dedicated IP

We took a chunk of Photojojo’s subscriber list, and began sending from a dedicated IP address on MailChimp’s server. When you setup a new IP, it takes some time to warm it up so that ISPs trust it (also see: Should you send from a dedicated IP?).

The Stats: What Happened After the Switch

As it turns out, their list size didn’t take much of a beating at all. It only shrank 3.3%.

Their first campaign did get the highest abuse complaint rate of all, but it was only 0.05%. Totally within industry averages. We’ve definitely seen worse. But by their 4th campaign, their abuse complaint rate was down to almost zero (0.009% to be exact — sheesh, I had to switch to a scientific calculator in order to handle all the decimal points).

Again, I normally expect about 40-50% of a list to get chopped off when people switch from an in-house system that they’d been using for many years. So Photojojo’s stats are simply amazing.

D’oh! Deliverability Dropped!

Notice how deliverability actually dropped at campaign #7? It fell to 79.7% to the inbox, 8.3% bulk, and 12.1% missing. That scared the bejeezus out of us. We ran to their Mailbox Monitor report, and discovered a few things:

  • First, the campaign (here’s a link to view its archive) got blocked pretty bad by a couple spam filters (Postini and MessageLabs) for its content. If you skim it, you’ll see words like "profitable, biz, fortune, win, and free." It’s easy to see why some spam filters might confuse this with a "get rich quick" kinda scam.
  • Second, Yahoo and SBC saw around 30% going to the spam folder. Digging through some email headers in our queue, we learned that it was due to content.
  • Third, one of the medium-sized ISPs blocked this campaign really hard because we weren’t throttling enough. And who knows, maybe the content played some role too. What’s interesting is that this particular ISP never popped up on our radar as a problem before, but Photojojo’s list happened to have a high concentration of subscribers on this ISP, so now it became a more noticeable issue. When they sent their next campaign (only 2 days after this one), deliverability went back up to 89%. Their content problems were gone, but that one ISP still gave us the same grief. Hmm. We had previously used our MTA software’s recommended settings for this ISP, but decided to tweak it a little ourselves. And it worked. By the next campaign, deliverability was back up to 99.7%. We took what we learned from this ISP’s throttling requirements, and applied it to the rest of our IPs (shared and dedicated), so that all of MailChimp’s customers could benefit from what we learned.

Obviously, it would be better to have zero deliverability issues, but the lesson here is that I’m not sure Photojojo would have the diagnostics, let alone the resources, to respond like we did to this ISP. When you consider how often ISPs change their rules to block spam, that’s one more question to ask yourself if you’re considering an ESP: "Could our own staff do this kind of thing in-house?"

Send Stuff People Actually Want. Repeat.

Looking back at the transition, the most amazing thing to me is how little Photojojo’s list size dropped.

In fact, their list has been growing at a nice clip ever since the switch.

My past experience has been that many email marketers get lazy after switching. As if all their email problems are solved now that they’ve outsourced "their email server." But when switching to an ESP, you’ve got to remember that we’re cleaning bounces and handling unsubscribes far more efficiently than you ever did on your own. So you’ll start to see attrition more than ever before. It was always there. You just never knew it.

And so a lot of email marketers are surprised at how much work it can be to keep your subscriber list growing. No, I don’t mean you have to spend a ton of money on co-reg or advertising or triggered campaigns.

Photojojo does it by maintaining a clean, permission-based list of people who actually want their content. Then, they send really, really good content. Over and over again. That’s a lot of work! Their newsletters are fun, well written, and best of all, actually useful. They also use Facebook and Twitter to keep the small-talk going in-between newsletters. As a result, Photojojo has some seriously loyal and engaged subscribers on their list, and they never have to worry about huge fluctuations in their list if they change servers.

Speaking of not worrying, this might be the most compelling reason to switch to an ESP. When I asked Amit from Photojojo how everything was going since the switch, he replied with this testimonial:

"So far, so great! The transition was remarkably smooth. Things are so busy for our business around the holidays, sometimes it seems like we spend the whole month just putting out fires. It’s a wonderful thing to know that our email is something we don’t have to worry about anymore."

Personally, I prefer off-the-cuff testimonials, so this is the part of Amit’s reply that I really loved:

"Seriously, things are great. The last three weeks have been crazy and it’s been nice to know that email isn’t something we need to worry about."

Somewhat related articles: