Contrary to popular belief, switching to an ESP (email service provider) like MailChimp, ConstantContact, etc., is not a silver bullet for deliverability.
Spam filters and email gateways look at two kinds of things: 1) your email message, and 2) your email infrastructure. In terms of your email message, ESPs can’t help it if you send spammy content, manage your lists like an idiot, or if a link in your email has a bad domain reputation. But as ReturnPath points out here, ESPs do have one important advantage: our infrastructure.
So if you’re managing your own email servers and delivery issues with ISPs, firewalls, throttling, feedback loops, and bouncebacks are becoming a supreme headache, you might be thinking about moving to an ESP.
This article is about how one email sender, Photojojo, decided to switch from their in-house solution to MailChimp. We’ll show you how we helped them make the transition, what we experienced, and how it impacted their deliverability.
I know you’re just looking for the delivery stats, so let’s just cut straight to the chase.
Here’s what happened to Photojojo’s deliverability after switching to MailChimp:
Keep in mind that this is being measured by using ReturnPath’s seed list across 13 weekly campaigns. A seed list is simply a bunch of email addresses setup at various ISPs (and ReturnPath’s seed list covers more than 130 domains including the top ISPs in the U.S., European, Canadian, Latin American and Asia Pacific markets.).
Anyway, as you can see, Photojojo’s inbox% eventually got pretty high (in the 99% zone). There were even a couple instances of 100% (again, this is based on 100% of emails on ReturnPath’s seedlist receiving the message, not some absolute math or anything).
You may also notice a few other things, so let me add my notes to the chart:
and I’ll explain those notes in a minute.
Setting Expectations for the Switch
But first, some backstory. Before they switched to MailChimp, we learned that Photojojo had been sending emails from their own in-house system for a few years. This is pretty impressive, considering their list is in the hundred of thousands. They were managing unsubscribes and bounces themselves, but not quite as automagically as an ESP can. So we warned them that when they switch, things wouldn’t be all good.
We set the following expectations:
- Initially, they’d probably see a dramatic drop in list size, since our bounce cleaner would clean out all those dead email addresses. My experience in these situations is that list sizes get slashed by as much as 50%. Yes, you read that correctly: 50%.
- Second, there might be a rise in spam complaints, if all of a sudden, old subscribers who weren’t receiving emails from Photojojo (because of delivery problems) actually started getting them.
In fact, our phone conversations kinda went like this in the beginning:
Me: Things are going to get worse before they get better, Amit. I’m warning you.
Amit: Yes, we understand there may be a decrease in our list size when we switch over. It’s to be expected.
Me: No seriously, you’re gonna hate us. You’re going to scream at me for ‘ruining your business.’
Amit: We know our bounce cleaning wasn’t great in the past, and we totally understand the repercussions. We’re fine.
Me: Yeeeeeeeah, right. We’ll see.
Preparing for the Switch
In an ideal world, we’d have a couple months to prepare for the switch. The reality was that we were only about 2 weeks (4 campaigns) away. With so little time, here’s what we recommended to prep Photojojo for the switch:
1. Warn all subscribers of the switch
We recommended they tell all current subscribers that Photojojo was about to switch email systems. Photojojo should ask subscribers to add the Photojojo email address to their address books to prevent delivery issues. So at T-minus 4 campaigns, their pre-header was changed to include this message:
they sent 3 campaigns with that header, and on the last one before switching to MailChimp, they changed it to a more urgent looking:
a similar “heads-up” message was placed at the top of their website as well. If I had half a brain, I would have taken screenshots to show you those. You’ll just have to take my word for it.
2. Measure deliverability now, to establish the baseline
We gave Photojojo our ReturnPath seed list so that we could measure the deliverability of their in-house solution before switching to MailChimp, and then again after. After all, we need to know if MailChimp is even helping anything, right? If you look at the graphs above, the first 4 campaigns that you see are before MailChimp, from their own servers. All subsequent campaigns are sent from MailChimp. You can see a gradual rise in their inbox % after switching. Whew!
3. Warmup their dedicated IP
We took a chunk of Photojojo’s subscriber list, and began sending from a dedicated IP address on MailChimp’s server. When you setup a new IP, it takes some time to warm it up so that ISPs trust it (also see: Should you send from a dedicated IP?).
The Stats: What Happened After the Switch
As it turns out, their list size didn’t take much of a beating at all. It only shrank 3.3%.
Their first campaign did get the highest abuse complaint rate of all, but it was only 0.05%. Totally within industry averages. We’ve definitely seen worse. But by their 4th campaign, their abuse complaint rate was down to almost zero (0.009% to be exact — sheesh, I had to switch to a scientific calculator in order to handle all the decimal points).
Again, I normally expect about 40-50% of a list to get chopped off when people switch from an in-house system that they’d been using for many years. So Photojojo’s stats are simply amazing.
D’oh! Deliverability Dropped!
Notice how deliverability actually dropped at campaign #7? It fell to 79.7% to the inbox, 8.3% bulk, and 12.1% missing. That scared the bejeezus out of us. We ran to their Mailbox Monitor report, and discovered a few things:
- First, the campaign (here’s a link to view its archive) got blocked pretty bad by a couple spam filters (Postini and MessageLabs) for its content. If you skim it, you’ll see words like “profitable, biz, fortune, win, and free.” It’s easy to see why some spam filters might confuse this with a “get rich quick” kinda scam.
- Second, Yahoo and SBC saw around 30% going to the spam folder. Digging through some email headers in our queue, we learned that it was due to content.
- Third, one of the medium-sized ISPs blocked this campaign really hard because we weren’t throttling enough. And who knows, maybe the content played some role too. What’s interesting is that this particular ISP never popped up on our radar as a problem before, but Photojojo’s list happened to have a high concentration of subscribers on this ISP, so now it became a more noticeable issue. When they sent their next campaign (only 2 days after this one), deliverability went back up to 89%. Their content problems were gone, but that one ISP still gave us the same grief. Hmm. We had previously used our MTA software’s recommended settings for this ISP, but decided to tweak it a little ourselves. And it worked. By the next campaign, deliverability was back up to 99.7%. We took what we learned from this ISP’s throttling requirements, and applied it to the rest of our IPs (shared and dedicated), so that all of MailChimp’s customers could benefit from what we learned.
Obviously, it would be better to have zero deliverability issues, but the lesson here is that I’m not sure Photojojo would have the diagnostics, let alone the resources, to respond like we did to this ISP. When you consider how often ISPs change their rules to block spam, that’s one more question to ask yourself if you’re considering an ESP: “Could our own staff do this kind of thing in-house?”
Send Stuff People Actually Want. Repeat.
Looking back at the transition, the most amazing thing to me is how little Photojojo’s list size dropped.
In fact, their list has been growing at a nice clip ever since the switch.
My past experience has been that many email marketers get lazy after switching. As if all their email problems are solved now that they’ve outsourced “their email server.” But when switching to an ESP, you’ve got to remember that we’re cleaning bounces and handling unsubscribes far more efficiently than you ever did on your own. So you’ll start to see attrition more than ever before. It was always there. You just never knew it.
And so a lot of email marketers are surprised at how much work it can be to keep your subscriber list growing. No, I don’t mean you have to spend a ton of money on co-reg or advertising or triggered campaigns.
Photojojo does it by maintaining a clean, permission-based list of people who actually want their content. Then, they send really, really good content. Over and over again. That’s a lot of work! Their newsletters are fun, well written, and best of all, actually useful. They also use Facebook and Twitter to keep the small-talk going in-between newsletters. As a result, Photojojo has some seriously loyal and engaged subscribers on their list, and they never have to worry about huge fluctuations in their list if they change servers.
Speaking of not worrying, this might be the most compelling reason to switch to an ESP. When I asked Amit from Photojojo how everything was going since the switch, he replied with this testimonial:
“So far, so great! The transition was remarkably smooth. Things are so busy for our business around the holidays, sometimes it seems like we spend the whole month just putting out fires. It’s a wonderful thing to know that our email is something we don’t have to worry about anymore.”
Personally, I prefer off-the-cuff testimonials, so this is the part of Amit’s reply that I really loved:
“Seriously, things are great. The last three weeks have been crazy and it’s been nice to know that email isn’t something we need to worry about.”
Somewhat related articles:
Very interestring article Ben, but I can’t help wonder? Is it the fact that their list in the “100s of thousands” which meant huge revenue for MC, get them this undivided attention?
What about a niche operation with a list of say 10,000?
Would you work with a company like ours to do the same if we switched over from our self-mailing capabilities? – which we’re not happy with anyway.
Mike, sometimes we just like to roll up our sleeves and get our hands dirty. Especially when there’s good case-study-potential like this one, where other customers can learn from it.
What is Throttling? In this context:
And Do you deal with all of the UK ISP’s
Third, one of the medium-sized ISPs blocked this campaign really hard because we weren’t throttling enough. And who knows, maybe the content played some role too. What’s interesting is that this particular ISP never popped up on our radar as a problem before, but Photojojo’s list happened to have a high concentration of subscribers on this ISP, so now it became a more noticeable issue. When they sent their next campaign (only 2 days after this one), deliverability went back up to 89%. Their content problems were gone, but that one ISP still gave us the same grief. Hmm. We had previously used our MTA software’s recommended settings for this ISP, but decided to tweak it a little ourselves. And it worked. By the next campaign, deliverability was back up to 99.7%. We took what we learned from this ISP’s throttling requirements, and applied it to the rest of our IPs (shared and dedicated), so that all of MailChimp’s customers could benefit from what we learned.
Jodie, let’s say you have a list of 5,000 people and 1,000 of them have @yahoo.com email addresses. Yahoo might not like it if we send all 1,000 to them at once. You have to “throttle” or slow them down a little. If you don’t, they think you’re a meanie. That’s kind of a gross oversimplification, but you get the gist. Furthermore, each ISP actually has different throttling limits, and they don’t all make their guidelines public. One more thing: ISPs will sometimes change those guidelines. For obvious reasons, ISPs don’t exactly make it easy to call them up and ask, “Howdy, I’m trying to send you a massive amount of email, so I’d like some tips on how to slide them right under your door and get them to the inbox.” Always keep in mind they’re not trying to frustrate legit senders; they’re dealing with a bajillion pieces of incoming spam every day.
So now that you know what throttling means, let’s take the remainder of your list, and chunk it up more. Maybe 700 of your subscribers are using Hotmail, and 600 are on Gmail. We have to deliver to those chunks a little differently than the others (which might shed new light on your campaigns’ domain performance reports).
Maybe one of those ISPs has severe throttling limits. So we have to take that chunk of subscribers and put them in another queue, and slooooooowly send those out. But what if that ISP is having some technical difficulties (say, during a holiday rush), and starts bouncing back all email, and telling us to try again later because they’re too busy? We have to take those and wait a while, then try again. Sloooooooooowly.
As you can see, there’s a lot of infrastructure-related stuff you have to deal with when delivering a massive amount of email. Once your lists get very large, most home grown solutions just won’t do.
Great article, we’ve been telling people for years that the single most important factor in sending high volume email is to build good history on a single or few IPs. If a prospective client asks if you can refresh IPs after a few months, red flags should go up! If you’re actually sending messages people want to receive, there should be no need to switch IPs. Bottom line is if the complaints are above one tenth of one percent, you’ve either got a bad list or the messages you’re sending are not what people want to receive.
My name is Brandon and I’m the deliverability engineer here at Mailchimp. Just wanted to introduce myself and hope to be able to answer or comment on any questions that might come up.
I’m a photog – you’re a nice looking gentleman. Give us a better pic… ;-)
hey…that’s a good article…i have been following mc since apr 09….since then the number of mc users/customers has grown from 175,000 to 225,000 (as per your homepage)…but the price plans have been the same…
i am no business geek, but with more customers wouldnt your fixed costs come down, as they are now spread over more customers….
y dont you revise your price plans and spread some monkey love to your international customers like me :)
Is there a deliverability difference in results from systems that use Pivotal Veracity or just from home grown solutions like the one you mention?
Hi Edward, sorry, I don’t think I totally understand what you’re asking. I’ll give this a shot, tho…
Photjojo’s “home-grown” solution that I described above was their delivery servers. They had no home grown deliver(ability) tracking. We used ReturnPath’s Mailbox Monitor to measure deliverability before & after their switch to MailChimp, and documented a significant improvement. Are you asking whether or not a home-grown deliver(ability) solution makes a difference vs something like PV? If so, my take on all this is that you can definitely just do your own deliverability tracking by setting up a bunch of test accounts at ISPs and creating your own seedlist. But services like ReturnPath and Pivotal Veracity typically build and maintain much more comprehensive seedlists than you ever could, plus they give you troubleshooting tips, ISP advice, and lots of details about your campaigns and why they might’ve failed.
Using a deliverability tracking solution (home-grown or not) just measures your deliverability. It doesn’t affect it. I mean, aside from any possible Heisenberg Observer Effects and all.
Is it possible for all MailChimp users to monitor their deliverability rates?
Ross, there are actually a lot of deliverability tools out there that you can use. It mostly just involves sticking their seed list into your subscriber list. A search on Google for “deliverability tools” should get you started. If/when you import a seedlist into your MailChimp account, I’d recommend giving the entries on the seedlist some kind of easy-to-find FNAME/LNAME. For example, SEED/LIST. That way, if you ever want to clean them from your list, you can do a search, export, then mass delete them.
Our deliverability has improved dramatically since we switched to Mail Chimp, which was basically our test to determine if this was a tool we would utilize long-term. Now we’ve begun integrating further, customizing automated emails, unsubscribe pages and posting to social sites. We recently created our first segment and conversion was through the roof. A/B tests are easy to setup and fun to monitor since Mail Chimp is constantly refreshing the data. All in all we’re happy with the monkey thus far and believe there is a lot of room for us to grow with this tool. I also want to share the customer support is fast and helpful. There was a glitch downloading lists of 50K or more (ours is 100K). When I brought it up they acknowledged the glitch existed. The next time I spoke with them the problem was resolved. :-)