It would be all fine and dandy if people would follow delivery status notification best practices and guidelines. But they don’t. Sometimes this is a reaction to spam, and sometimes it’s just ignorance.

For example, some server admins insert snarky messages in their email headers, like “We don’t want your message. If you send email to us again, we’ll report you.” Well, that’s their prerogative and all, and we’re happy to never send to them again, but if they simply hard bounced the email, we’d be able to clean it from the list faster.

Then there are some ISPs who are downright deceptive with their bounceback codes…

They’ll send back a bounce that tells us that the intended recipient doesn’t exist. But look closely at their bounce headers, and you see little messages like, “but if you wait a few hours and try again, it’ll get through — wink wink.” An interesting way to tell if there are humans sending the email.

Then there’s the problem of “silent dropping:”

“As discussed in Section 7.8 and Section 7.9 below, dropping mail without notification of the sender is permitted in practice. However, it is extremely dangerous and violates a long tradition and community expectations that mail is either delivered or returned. If silent message-dropping is misused, it could easily undermine confidence in the reliability of the Internet’s mail systems. So silent dropping of messages should be considered only in those cases where there is very high confidence that the messages are seriously fraudulent or otherwise inappropriate.” Source: wikipedia

And there have been cases where an ISP will temporarily go down for hours (or days), and in the meantime, they send you back hard bounces or erroneous “you’ve been blocked” reports. Should you clean those hard bounces from your list? Technically, it’s a “best practice.” But clearly, the ISP was broken when you sent. Hardware just breaks sometimes (See: Not All Delivery Problems are Spam Related).

Some receiving servers have sent back hard bounced messages that were intended for recipients that we know exist, because we have double opt-in evidence, and open/click actiivtiy. We find out about these problems when recipients complain to the sender about not getting the email they requested, and the sender escalates it to us, and then we trace it back to the recipient’s IT guy setting up “custom” rules. To be clear, it’s their prerogative to setup their custom rules. We don’t hold it against them (spam’s ruining it for everyone). But this does create a problem that requires a custom solution of our own.

Good Deliverability Depends on Proper Bounce Management

See why bounce cleaning can be frustrating? No wonder people who try to manage their email marketing in-house see such dramatic improvements in deliverability when they switch to an ESP (case study). They’re usually unable to properly clean the bounces from their lists.

So they don’t.

And if you keep sending messages to non-existent accounts, ISPs will block you because you look like a spammer who purchased an old email list.

Improper bounce cleaning can seriously damage your domain reputation.

Also, we’re seeing new trends in the way our customers send emails. People are automating more with RSS-to-email, and via our API. More daily senders with extremely large lists (daily deals, mobile apps, location-based check-in services, etc) are depending on us to get their emails delivered, but also depend on us to intelligently manage those lists. Simplistic bounce cleaning rules, combined with deceptive bounce errors, can result in their lists shrinking faster than new members can opt-in. This, in turn, often results in irrational behavior by the sender (purchasing lists, using bad/old lists, un-bouncing everybody, ESP-hopping with old, uncleaned lists, and on and on).

So we’re tweaking the way we handle bounces.

Our strategy for a long time now has been to perform deep, ongoing analysis of bounce headers in order to create “the most insanely thorough bounce back interpreter holy-grail known to man” (and we usually end that statement with an evil, nerdy laugh). And we’ve come a long way with that approach.

Moving forward though, MailChimp will be factoring engagement activity into our bounce cleaning decisions (read about how MailChimp measures engagement).

For example, if we send an email and a receiving server tells us that a recipient “does not exist,” but we have open and click activity in the last 45 days to prove otherwise, we’re not going to blindly clean that recipient from the list. We know they exist, and we know their account works, so we’re going to give them a few more chances than we normally do. If, however, we see that there’s very little (or no) activity by that recipient, we clean them under the same rules we’ve used in the past.

We’re not going to get into specifics about how many stars justifies a “clean vs. a keep,” or exactly how many chances we give hard and soft bounces. The algorithm will surely be adjusted and tweaked over time. The point we’re trying to make is that email is evolving faster than ever (thanks to changing social and mobile behaviors of recipients and senders), and MailChimp is adapting and innovating along with it. Even in the very un-sexy area of bounce management.

About 240,000 of them are “old” (inactive) subscribers. About 70,000 are relatively “new” (active) subscribers.

They recently segmented their list and sent the same newsletter to each group (separately) over the same IP address, about 6 hrs apart from each other. Around 2pm, they sent the newsletter to the large, inactive list. Around 8pm, they sent the same newsletter to the active list.

The results are eye-opening…

Stats for the “Inactives” list (241,832 recipients):

Spam Complaints: 43
Open Rate: 6%
Click Rate: 2.4% (and 7,688 total clicks)
Unsubscribes: 264
Bounces: 6,878 (2.8%)

Stats for the “Actives” list (69,642 recipients):

Spam Complaints: 3
Open Rate: 36.3%
Click Rate: 7.4% (and 6,925 total clicks)
Unsubscribes: 96
Bounces: 128 (0.18%)

The number of bounces for the “old-members-list” is what I’d call very high. The number of abuse reports (43) is what I’d call disturbing. These are humans clicking the “this is spam” button for a run-of-the-mill company newsletter.

More interesting is what happened to their deliverability for the old list. We measured it using ReturnPath’s Mailbox Monitor service. Here’s a screenshot:

As you can see, their emails went 100% missing from AOL and Hotmail, 80% from Yahoo, and 70% for Comcast.

Also, it’s almost a day later, and 20% of the “sent-to-old-members” list still hasn’t been delivered to Yahoo, because they keep getting deferred.

My guess is the sheer number of spam reports from the old list is what got them blocked, and therefore their emails have gone missing (here’s some explanation from AOL on how they measure IP reputation).

If you’ve got a bad reputation, you’ll get blocked by AOL (and most of the major ISPs). Comcast’s Postmaster lists how emails get throttled based on your “Sender Score.” The lower your score, the slower you’ll get delivered.

So how is your score determined?

AOL measures your IP reputation by:

• Spam complaints (from humans clicking the “spam button”)
• Spam filter triggers
• Bounces from undeliverable addresss (i.e. old lists)

I wish we had deliverability stats to show you how the “active-members-only” went, but we forgot to track the good one (d’oh). Maybe next time.

Even without a side-by-side deliverability comparison, the end result is clear: old lists suck.

They generate too many abuse complaints, which can get your entire campaign blocked by ISPs. Then, the active members who want your emails never get your message.

Related Stories:

• Desperation Marketing (see “living dead”)
• Why your email won’t get delivered
• ReturnPath’s Deliverability Best Practices Guide (PDF)
• Effects of old email lists (an oldie but goodie)

Ever thought about segmenting your email campaigns so that they only go to the “active” members on your list? Here’s how to do that in MailChimp.