So let’s take a look. The first two charts in our “Quick Insights” shows the email opens based on time of day and day in the week. The ‘y’ axis displays the percentage of opens on our tested data. This is an average per campaign rate and what the campaign can expect. It may look simple, but it’s very useful. (Scratching head) “So, what you’re telling me is Thursday around 3 pm is an awesome time to send my Morning Coffee Newsletter?” Not really. Keep in mind that you should use this data as a reference. You certainly don’t want to send your Morning Coffee Newsletter at 3 pm. That would be silly, but then again…some of us need a little afternoon pick me up.

(click on the images for a closer look)

Well, how can I really use this then? Say for instance you don’t send a Morning Coffee Newsletter and you normally send your campaign out at 10 pm. We encourage you to try something new and set up a Time Warp campaign and set it for example, at 3 pm. Or even create an A/B split campaign  based on your different campaign delivery times. See how it works for you. Ultimately, you want to send your campaigns where you see fit but, you may come to find that based on your industry, A/B split test results, or time warp campaign, that a different delivery time works better for your campaigns.

Next on the menu is “The Effects of Subscriber Recency on Open /Click Rates.” Recency is a fancy word for: occurring right before the present. So, when your subscribers sign up for a newsletter they are likely to be more engaged from the start. Again the ‘y’ axis presents the percentage of opens across our tested data. You might think its a bit odd to see a dwindling of engagement as time goes on however, dependent upon your industry and what type of campaigns you send, you can fine tune that engagement. If you start experiencing depleting engagement, it may be time to consider reengaging your subscribers.

Last but not least is the “Effects of Number of Links in Email on Click Rate.”
The ‘y’ axis presents the percentage of click rates across our tested data. As this graph shows, the more links you have, the more clicks you are likely to have. This is true, but remember it’s not always the case and could effect your engagement. We advise that the links are relevant and pertain to your campaign content. It is also important to minimize the use of shortened links and check for misspellings. If you use URL shortening, have a look here.  A healthy balance of links and general campaign content is always good. So don’t overload your subscribers with hundreds of links, as they may eventually stop opening emails in fear of a “total link assault.”

All in all, we want to provide you with rough benchmarks of our performance and what you can expect from your campaigns. Every data point should work symbiotic with each other to achieve optimal campaign performance. What you send matters to you and to MailChimp, which is why we created Chimp Charts. Now off to eat some banana bread!

The first few comments we got were understandably furious. But eventually, the conversation changed. We think we were on the way to a very constructive discussion. I really enjoyed the dialogue I had with people offline as a result of all this, and I want to thank all the email companies who commented here — CritSend, CampaignMonitor, PostMarkApp, and Al Iverson’s A-1 Super Awesome Home DSL Email Service.   I mean, don’t get me wrong. We’re competitors. We’re not going to be singing Kumbaya around the campfire with each other any time soon. It’s just nice talking to people who know their stuff. I wish the discussion could continue.

But my patience has been worn down. ReturnPath is naggi—asking me politely to take this post down, because they “don’t want to arbitrate arguments between their partners.” [I didn't realize we were asking them to arbitrate] I suggested that, as an independent, unbiased scoring system, they should just do what I do: ignore the bastards. Actually, I suggested the complainers needed to “grow some” and that ReturnPath ought to tell them so. But that’s not how ReturnPath rolls (thankfully, I guess).

Anyway, some of the arguments we heard about our posted methodology seemed to go like this: “Your methodology is flawed, because SenderScore penalizes IP addresses that send very low volumes, and that don’t have a high reputation. For example, I have an IP address that has GREAT inbox rates (um, trust me) but that have a low SenderScore.”

Well, yes. We know that low-volume, low-reputation IPs can get great deliverability.

Buuuut we happen to think that an ESP’s very job is to send high volumes of email while simultaneously maintaining a good IP reputation. We send out tons of email through our infrastructure, 24/7. If it were a race car engine, SenderScore’s our tachometer. Does it show actual vehicle speed? No. But it’s extremely indicative of engine performance. If we see our SenderScore drop from 94 to 70, there’s a problem with the engine. It’s time to pull over and get that deliverability fixed.

So to people who say “SenderScore is a bogus number” we respectfully disagree. It may not work for all senders, but it works for ESPs. The ones who send high volume. And want to measure their reputation.

Obviously, I am not concerned with what other ESPs think, or how they respond. But personally, I think that ReturnPath’s naggi — um, polite requests for me to pull this blog post is actually going to work against them. I think they’re defending the very people who are disputing the validity of SenderScore. On the one hand, that concerns me. On the other hand, I’ve always appreciated irony. And I absolutely hate the blinking red voicemail light on my office phone.

So this awesome blog post, which attempted to highlight the usefulness of ReturnPath’s SenderScore, is now officially yanked — at the request of ReturnPath.

It would be all fine and dandy if people would follow delivery status notification best practices and guidelines. But they don’t. Sometimes this is a reaction to spam, and sometimes it’s just ignorance.

For example, some server admins insert snarky messages in their email headers, like “We don’t want your message. If you send email to us again, we’ll report you.” Well, that’s their prerogative and all, and we’re happy to never send to them again, but if they simply hard bounced the email, we’d be able to clean it from the list faster.

Then there are some ISPs who are downright deceptive with their bounceback codes…

They’ll send back a bounce that tells us that the intended recipient doesn’t exist. But look closely at their bounce headers, and you see little messages like, “but if you wait a few hours and try again, it’ll get through — wink wink.” An interesting way to tell if there are humans sending the email.

Then there’s the problem of “silent dropping:”

“As discussed in Section 7.8 and Section 7.9 below, dropping mail without notification of the sender is permitted in practice. However, it is extremely dangerous and violates a long tradition and community expectations that mail is either delivered or returned. If silent message-dropping is misused, it could easily undermine confidence in the reliability of the Internet’s mail systems. So silent dropping of messages should be considered only in those cases where there is very high confidence that the messages are seriously fraudulent or otherwise inappropriate.” Source: wikipedia

And there have been cases where an ISP will temporarily go down for hours (or days), and in the meantime, they send you back hard bounces or erroneous “you’ve been blocked” reports. Should you clean those hard bounces from your list? Technically, it’s a “best practice.” But clearly, the ISP was broken when you sent. Hardware just breaks sometimes (See: Not All Delivery Problems are Spam Related).

Some receiving servers have sent back hard bounced messages that were intended for recipients that we know exist, because we have double opt-in evidence, and open/click actiivtiy. We find out about these problems when recipients complain to the sender about not getting the email they requested, and the sender escalates it to us, and then we trace it back to the recipient’s IT guy setting up “custom” rules. To be clear, it’s their prerogative to setup their custom rules. We don’t hold it against them (spam’s ruining it for everyone). But this does create a problem that requires a custom solution of our own.

Good Deliverability Depends on Proper Bounce Management

See why bounce cleaning can be frustrating? No wonder people who try to manage their email marketing in-house see such dramatic improvements in deliverability when they switch to an ESP (case study). They’re usually unable to properly clean the bounces from their lists.

So they don’t.

And if you keep sending messages to non-existent accounts, ISPs will block you because you look like a spammer who purchased an old email list.

Improper bounce cleaning can seriously damage your domain reputation.

Also, we’re seeing new trends in the way our customers send emails. People are automating more with RSS-to-email, and via our API. More daily senders with extremely large lists (daily deals, mobile apps, location-based check-in services, etc) are depending on us to get their emails delivered, but also depend on us to intelligently manage those lists. Simplistic bounce cleaning rules, combined with deceptive bounce errors, can result in their lists shrinking faster than new members can opt-in. This, in turn, often results in irrational behavior by the sender (purchasing lists, using bad/old lists, un-bouncing everybody, ESP-hopping with old, uncleaned lists, and on and on).

So we’re tweaking the way we handle bounces.

Our strategy for a long time now has been to perform deep, ongoing analysis of bounce headers in order to create “the most insanely thorough bounce back interpreter holy-grail known to man” (and we usually end that statement with an evil, nerdy laugh). And we’ve come a long way with that approach.

Moving forward though, MailChimp will be factoring engagement activity into our bounce cleaning decisions (read about how MailChimp measures engagement).

For example, if we send an email and a receiving server tells us that a recipient “does not exist,” but we have open and click activity in the last 45 days to prove otherwise, we’re not going to blindly clean that recipient from the list. We know they exist, and we know their account works, so we’re going to give them a few more chances than we normally do. If, however, we see that there’s very little (or no) activity by that recipient, we clean them under the same rules we’ve used in the past.

We’re not going to get into specifics about how many stars justifies a “clean vs. a keep,” or exactly how many chances we give hard and soft bounces. The algorithm will surely be adjusted and tweaked over time. The point we’re trying to make is that email is evolving faster than ever (thanks to changing social and mobile behaviors of recipients and senders), and MailChimp is adapting and innovating along with it. Even in the very un-sexy area of bounce management.

I thought they handled it well, but that’s not why I’m writing. Those guys know what they’re doing. I’m writing because while reading through their technical details, I stumbled upon this snippet about email deliverability…

email verification irony
We’ve also discovered that a lot of the verification emails we’ve been sending out haven’t been going through. It seems that the mail server admins at some popular domains (e.g., comcast.net, rr.com, adelphia.net, and me.com) have their servers configured to consider all mail from reddit to be spam. This is because Trend Micro has marked Amazon’s entire EC2 network as a “dial-up pool”, and the aforementioned domains subscribe to Trend Micro’s list and block all mail from anyone on said list. We’ve written to Trend Micro explaining that we’re actually neither a spammer nor an individual end user, but rather an honest website that’s kind of a big deal, and they sent us a form letter explaining how to configure Outlook Express and encouraging us to ask our ISP for further information. We’ll try to figure something out as soon as time allows.

Yep. Welcome to our hell. Email delivery (on a massive scale) is kinda complicated.

Should anyone else out there have to setup an email delivery server all by yourself, this very topic (cloud-based MTAs) is covered in our guide:

Email Delivery for IT Professionals on page 5, under Hosting and Hardware.

…And you don’t want your MTAs to live in the cloud or in a virtual  environment. MTAs need bare metal and fast drives. An MTA’s activity  is bound mostly to disk and CPU, so cloud and virtual environments  aren’t suited well to the task. Consider this when you’re thinking about  your network topology and designing your application. Your sending  infrastructure (or at least your MTA) may have to exist in another  datacenter. We’ve seen the effects of tying your sending infrastructure  in the cloud, and the results are poor performance and reputation.

Amazon EC2 is great for massive computing in the cloud. We’ve used it to run 61 trillion data comparisons for Omnivore. But it’s not-so-great for mass email marketing.

In case anyone out there thinks that was kinda preachy sounding of me, trust me: we only know this stuff because we’ve tried it already (not because we’re any smarter).

While on the topic of MailChimp’s free guides, I’d like to take this opportunity to point out our spiffy new MailChimp Resource Center. Look at all those redesigned covers! We hope you’ll agree it’s not your average whitepaper directory.

But diagnosing the exact reason your email was blocked can be extremely difficult.

The only way to really figure out why your message was blocked is to systematically test each variable: change your subject line, and send another test email. Go check all your test accounts. Did it get blocked again? Well, change this link. Still blocked? Change another link. Wasn’t your links? Swap out the images. Not it? Change your content. Over and over, till you find the culprit. Then, do all that again for the next spam filter. Complete p.i.t.a.

So we automated all that with our new Delivery Doctor tool. Push one button, and we’ll automagically slice and dice and analyze your email and run dozens of tests until we find the root of your block.

Then, we tell you what to fix…

Delivery Doctor's Diagnosis of your problem

You’ll find the Delivery Doctor at the bottom of the Pre-Delivery Checklist screen (the last step before sending out your campaign).

Click the button, and we’ll start analyzing. Depending on how many tests we have to run, it could take a while. So we send you the results via email. Please don’t sit there and watch the spinning animation (if you really enjoy spinning animations, this one’s much more fun).

Keep in mind that if you have a problem with your email, and you run the Delivery Doctor, we actually send this to real accounts (setup with consumer ISPs and B2B filters). We test for common issues, like subject lines, links, email content, images, and more. This means that, depending on the nature of your problem, as many as 44 different tests might be run on your email.

Because it’s such an intense diagnosis, we can’t offer this to everybody. It’s totally free to use, but is only available to users on paid plans (not to our freemium users). Also keep in mind that this tool is still in beta. We’re already seeing hundreds of tests being run, just hours after launching it, and due to heavy loads we’ve had to remove a few of the tests we perform (slow down, people!). It’s beta. In due time, we’ll add more test criteria and really refine the product.

In general, the more stars, the better your deliverability will be. So what’s your score?

Mailman Steve Padgett, age 58, stood before a Federal Court judge recently to receive his sentence. The crime? Delaying and destroying the very mail he was supposed to be delivering– third class mail, or more commonly, the JUNK.

This spring, authorities were contacted by a utility worker who noticed what appeared to be an excessive amount of mail piled at Steve Padgett’s home in Raleigh. When postal authorities went to investigate, they discovered third-class mail stacked in Padgett’s garage and buried in his lawn.

According to Padgett’s attorney Andrew McCoppin, it wasn’t a conscious stand against waste or a junk mail protest that spurred the mailman to hold onto the mailers. Rather, it was the inability to meet the demands of a job in a growing part of the county while contending with heart problems and complications from his diabetes.

Padgett was given probation, fined and also sentenced to 500 hours of community service.  And as a way to express our support for Mailman Steve and his junk mail minimizing tactics, MailChimp has helped bail him out by contributing to a fund that will cover Padgett’s fines.

How does this relate to email marketing you ask? Mailman Steve was keeping the spam out of people’s physical mailboxes, in the same way that MailChimp works to keep it out of your inbox. By taking simple steps like creating a good permission reminder and adhering to proper emarketing etiquette, you can take steps to ensure your email’s relevance and deliverability.

About 240,000 of them are “old” (inactive) subscribers. About 70,000 are relatively “new” (active) subscribers.

They recently segmented their list and sent the same newsletter to each group (separately) over the same IP address, about 6 hrs apart from each other. Around 2pm, they sent the newsletter to the large, inactive list. Around 8pm, they sent the same newsletter to the active list.

The results are eye-opening…

Stats for the “Inactives” list (241,832 recipients):

Spam Complaints: 43
Open Rate: 6%
Click Rate: 2.4% (and 7,688 total clicks)
Unsubscribes: 264
Bounces: 6,878 (2.8%)

Stats for the “Actives” list (69,642 recipients):

Spam Complaints: 3
Open Rate: 36.3%
Click Rate: 7.4% (and 6,925 total clicks)
Unsubscribes: 96
Bounces: 128 (0.18%)

The number of bounces for the “old-members-list” is what I’d call very high. The number of abuse reports (43) is what I’d call disturbing. These are humans clicking the “this is spam” button for a run-of-the-mill company newsletter.

More interesting is what happened to their deliverability for the old list. We measured it using ReturnPath’s Mailbox Monitor service. Here’s a screenshot:

As you can see, their emails went 100% missing from AOL and Hotmail, 80% from Yahoo, and 70% for Comcast.

Also, it’s almost a day later, and 20% of the “sent-to-old-members” list still hasn’t been delivered to Yahoo, because they keep getting deferred.

My guess is the sheer number of spam reports from the old list is what got them blocked, and therefore their emails have gone missing (here’s some explanation from AOL on how they measure IP reputation).

If you’ve got a bad reputation, you’ll get blocked by AOL (and most of the major ISPs). Comcast’s Postmaster lists how emails get throttled based on your “Sender Score.” The lower your score, the slower you’ll get delivered.

So how is your score determined?

AOL measures your IP reputation by:

• Spam complaints (from humans clicking the “spam button”)
• Spam filter triggers
• Bounces from undeliverable addresss (i.e. old lists)

I wish we had deliverability stats to show you how the “active-members-only” went, but we forgot to track the good one (d’oh). Maybe next time.

Even without a side-by-side deliverability comparison, the end result is clear: old lists suck.

They generate too many abuse complaints, which can get your entire campaign blocked by ISPs. Then, the active members who want your emails never get your message.

Related Stories:

• Desperation Marketing (see “living dead”)
• Why your email won’t get delivered
• ReturnPath’s Deliverability Best Practices Guide (PDF)
• Effects of old email lists (an oldie but goodie)

Ever thought about segmenting your email campaigns so that they only go to the “active” members on your list? Here’s how to do that in MailChimp.

According to George Bilbrey, AOL, Gmail, and Yahoo have implemented DKIM email authentication.

For those of you who don’t know, authentication is a way to prevent email forgeries, and it can improve your deliverability. The AOTA says Authentication has hit its “tipping point.”

Since 2004, when email authentication started to really pick up steam, ISPs have been testing (and changing) their support for authentication. Some use one method for inbound, and another for outbound. Some support all methods of authentication. Some have abandoned it, or are in limbo. It can be confusing.

So we compiled a chart of which ISPs are using which authentication methods over here.

Of course, we’ve also made authentication “MailChimp Easy:” Our customers can authenticate their email campaigns with one simple click, and we cover all the major authentication standards (DKIM, Domain Keys, SenderID, and SPF).

“The adoption of e-mail and domain authentication has reached its tipping point, exceeding 50% in several key metrics, according to the Authentication and Online Trust Alliance. The report found that 51% of the Fortune 500′s consumer-facing brands, 52% of the Fortune 500′s consumer-facing financial service brands and 54% of the top 300 brands in the Internet retailer segment are all using some form of e-mail authentication.”

You can read more about Authentication and how it affects consumer trust over at DMNews.

If you’re a MailChimp customer, you can authenticate all your email campaigns, just like the Fortune 500 brands do. It’ll make your emails look more trustworthy, and it’ll sometimes help you get through corporate email firewalls. And you can do this with one simple click. Here’s how to activate it (it’s free for all MailChimp customers).