Oct 3, 2006

Monitoring Deliverability

Spam filters are making it really hard to get emails delivered. In this latest email deliverability study from ReturnPath, 19.2% of legitimate emails they monitored never made it to the inbox. This study over at Pivotal Veracity says about 15% of emails get blocked. Yikes.

Do you have any idea how many of your emails are making it into your recipients’ inboxes? The only way to tell is to setup a TON of email accounts with all the major ISPs, send your campaign, then check each account one by one. Poking your eye with a hot needle would be more fun.

Luckily, you can just sign up for a deliverability monitoring service. They’ll do all the mundane work for you. I just recently signed up with one, and tested a MailChimp campaign…

There are quite a few services to choose from. A quick Google search turns up:

We went with DeliveryMonitor.com, because they were the easiest to jump right in and get started. I hear SenderScore and Pivotal Veracity are extremely thorough, but no free trials are readily available. DeliveryWatch looked very promising, but they don’t take American Express yet.

Here’s how these services typically work:

  1. They create a "seed list" of email addresses that you add to your subscriber list.
  2. The seed list is composed of email accounts they’ve setup with all the major ISPs
  3. For every campaign you send, you add a little tracking code somewhere in your content. It can be embedded so that recipients don’t see it.
  4. After you send your campaign, you can check back in a couple hours to see how you did (screenshot below).

You’ll see which ISPs let your emails through to the inbox, and which ones sent you to the spam folder (and which ones just lost your email outright). They’ll also check major blacklisting services and tell you if you were blocked by any of them.

When we tested the service, we sent a campaign to our own MailChimp customers, and we used the same range of IP addresses as all our "new" user accounts (none of our  "reserved" or "trusted-customer" IPs were used). Below is a screenshot of the graph that DeliveryMonitor gave us.

Some tips:
I’ve spoken with a couple people who run these services, and they gave a couple tips for people who are just getting into delivery audits:

  • Results are regional, meaning if you send lots of emails to customers in Europe, go with a provider who has a seed list with European ISPs (DeliveryWatch is one).
  • Don’t try to use these services on small/test campaigns. They need you to send to your actual list, because ISPs behave differently when you’re sending a small test vs. a real campaign to thousands of recipients on their network. That means you don’t send, tweak, send, tweak. You slowly gather data over several "real" campaigns and make changes accordingly.
  • Most of them charge a ‘per event’ fee. You pay each time you run a test. Sounds like it can get expensive, but think about what a royal pain it would be to actually setup a bazillion test accounts and then run reports on each one. DeliveryMonitor charges a monthly fee, with a monthly limit on tests.

Here’s the report we got from our campaign (click to zoom in). You can see that our biggest problem was with AOL. Seems 20% of our emails to their AOL recipients got spam filtered. This isn’t enough to alarm me, but if it were significantly higher, I’d make adjustments to my email content. If that didn’t work, I’d contact the AOL postmaster for help, or look into some accreditation services, like Goodmail or SenderScore.

If you’ve ever hit your send button and wondered, "how many of my emails are actually making it through?" you should try one of these services. It can be a huge time saver, and the information you get back can be extremely valuable (especially if you do a lot of e-commerce via email).