Mar 6, 2009

Your list activity score and deliverability

In MailChimp v4.1, launching later this month, we’ll be introducing something new to the List Dashboard.

Each list that you manage in MailChimp will have its own "List Activity Rating." Below is a mockup of what it’ll look like:


Told you we’re suckers for stars.

Your list activity score is based on an algorithm that tells us how "active" and "engaged" the members of  your list are. Generally speaking, the more active your list is, the better your deliverability (here’s what happens if you send to a list that’s not active). And since our job as an ESP is to maximize your deliverability, we need to know whose lists are active, and whose lists are not. Your list activitiy score will determine how we deliver your emails from our system.

Some background information

For the last 7 months or so, we’ve been doing some secret, "behind the scenes" research by scanning all campaigns sent from our servers.

We’ve been experimenting with the concept of "user activity."

Basically, what if we could tell how "active" or "inactive" your subscribers are? And how can we use that to determine how "active" your entire list is?

Armed with that knowledge, how could we improve the overall deliverability of our system?

For example, if you send an email campaign, and one of your subscribers has never, ever clicked or even opened any of your previous campaigns, that’s not a very active user. And what if a big chunk of your list is composed of similar "inactive" members? That’s probably an indication that you’ve got a bad/old list mixed in there. We’re not saying you’ve done something wrong, or that you’re a spammer. But your list is old. We won’t punish you for that (unless our abuse team thinks you’re breaking our terms of use). But knowing you have an old list, should we allow you to send to that list across the same IP addresses used by other MailChimp customers, who have extremely active/clean lists?

How we determine "list activity"

It’s not just about opens and clicks. We look at the number of bounces across your list, the number of spam complaints, and the number of unsubscribes.

Your list activity is an overall score based on the "activity" of each member in your list. Here’s a gross over-simplification of how it works (we’ll post more details later).

Let’s say John is one of your subscribers. John starts with a certain neutral score. John opens an email. John gets a +1. John clicks a link in your email. +2. You send a few campaigns, and John soft bounces. -1. Next campaign, John opens. +1. Mothers Day rolls around, and your boss tells you that "we really need to blast out something totally irrelevant to our list, or we won’t meet our quota." You have no spine, and you agree to send John something he didn’t give you permission to send him. John clicks the spam button. He gets a -2 and gets removed from your list.

Now imagine that measurement happening across your entire list of members. Average them all up.

That, roughly speaking, is your overall list activity score.

Double opt-in is not perfect.

In the past, we separated our IP ranges based on whether or not a customer used double opt-in lists, vs. single opt-in lists. The problem with that is the sheer number of single opt-in lists dwarfs the number of double opt-in lists. Also, those who use double opt-in lists tend to be occasional senders, not high-strung, "gotta have bazillions of subscribers and blast an email out every week" kind of people. I’m an occasional sender, which is why I don’t care one bit if a certain percentage of signups don’t complete the double opt-in confirmation process. Hey, the must not have wanted it that bad. Nothing wrong with that. Plus, I just prefer to send emails when I have useful stuff to say. Which is why I usually send quarterly(ish).  The problem, if you pool a bunch of us into a range of IPs together, is that very little email volume goes out. No email volume means no history. No history means no reputation (here’s an article we posted on the reputation of new IPs). If you get a sudden spike in volume, the lack of reputation can actually hurt deliverability.

Human reviewers are not perfect.

We also have a human review team here at MailChimp (I’m part of it too, sometimes). We’ve developed a "sixth sense" when it comes to reviewing new signups. We can look at their contact information, and in the blink of an eye, determine whether or not they’ll be a) a good user, b) an ignorant and problematic user, or c) kicked out the door. At some point, I documented something like 25 different criteria I use to judge someone. For example, if you sign up for a free trial with MailChimp, and you used ALL CAPS in every single field of our signup form, that’s not a good sign. If you don’t have a favicon on your website, you’re not a details kinda guy. If you don’t have a website at all, something’s fishy.

The human review team also had the power to "promote" certain users to a "trusted" IP range, away from the big shared pool. This decision would be based on whether or not the sender had a good history with us. The problem with that approach is you’re basing things on the customer.

Not the way they behave.

And customers change. Especially in desperate economic times. We saw that in the dot-com fallout days, and we’re seeing hints of it now.

I’ve approved some great customers who later did some stupid things. One customer who works for a very large corporation, with a near-perfect email history, recently volunteered for a local non-profit and imported their stale 4-year old list. It got one of our IPs blacklisted. Good user. Stupid behavior. Our abuse team can get our IP delisted, and restore our reputation relatively easy. His reputation? Maybe not so easy. And now, we will change the way we deliver his emails in the future, until he builds his score back up.

Moving forward, the system will judge lists based on their behavior, not by the reputation of their owners.

Stellar list scores = stellar deliverability

We’ve been running this for 7 months behind the scenes, and so far it’s been great. We’ve been comparing our list activity scores to overall deliverability (using ReturnPath’s Mailbox Monitor), and it’s spot-on.

Those with stellar list scores get stellar deliverability.

Every list on our system now has a list activity score, and in late March they’ll be revealed in your account.

Again, don’t think of this as a way to take people with "bad emails" and throw them into some swampy black hole of IPs. That would hurt MailChimp as much as it would you.

This system is not meant to punish anybody (our abuse team is working on that system. seriously). This is meant to show you how relevant your email marketing is, and how clean your list is. No matter what your score is, MailChimp’s deliverability will be great. But if you’re sending relevant email campagns to permission-based email lists, your deliverability will be better than great.

Think of this as an incentive. To do good.