In January, we announced Omnivore, our massive anti-spam research project that ran 61 trillion email data comparisons using genetic optimization algorithms in order to teach our network how to automatically detect and prevent abuse.

For those of you who don’t know, we built Omnivore in order to prepare for our big Freemium plan that we launched back on September 1st, 2009. We didn’t want to offer a free email marketing service without having a scalable system in place to protect our deliverability (not to mention the sanity of our Compliance Team).  Good thing, too.

In just under a year, MailChimp grew from 85,000 users to over 430,000. We couldn’t have grown 5-fold like that without Omnivore.

Here’s an update on what we’ve learned so far…

Since September 1st 2009 Omnivore has:

• Issued 69,927 warnings to 24,119 users for exhibiting bad behavior. Warnings like, “Hey, we detected a lot of unsubscribes from that last campaign — if it continues, we’re going to have a deliverability problem.” Another warning example would be something like, “Whoah, that last campaign had a higher than normal bounce rate. Something’s going on with your list hygiene. Here are some tips for you to address that issue before it gets worse.” Warnings usually never need a reply. They’re simple observations that let you know something’s wrong, and if it’s not corrected, could lead to more issues, which could lead to suspension.
• Automatically suspended 8,770 users. This typically happens when Omnivore sees something really alarming, and just can’t allow an email to leave our system. When a user’s account gets suspended, it’s placed into a queue for human review. Our compliance team basically investigates to see if it was a false positive, sends tips to users if it was an innocent mistake, or in some cases, might decide to permanently shut down the sender.
• Of those suspended accounts, 1,879 ultimately had to be shut down. Shutdowns don’t always mean the sender was evil. Sometimes they just mean that a sender might be sending content that’s too risky, and receives more than the normal amount of delivery problems or abuse reports. Even if they’re totally innocent, they can still cause harm, and we have to shut them down.

When I first wrote about Omnivore, I was very careful to explain that it was new, and still had a lot to learn.

Over the last few months, it’s learned a lot.

New Shades of Gray

Our approach with Omnivore has been primarily to look for those things that spam filters don’t. For example, we could simply scan outgoing email with Spam Assassin’s criteria, and block offending messages. But that would only catch the “black and white” stuff. That’s fine for catching the horrible appendage-enlargement spam we’re all so familiar with. But ESPs deal with “ignorant spam” more than “evil spam.” Ignorant spam is harder to define. It’s a gray area.

Speaking of shades of gray:

And that’s the stuff Omnivore looks for.  Stuff that looks like perfectly legitimate business mail, and that would slip past most spam filters, but then generate a ton of spam complaints from recipients (traits that humans think are spammy, but that spam filters miss).

Since launching, we’ve discovered even more shades of gray in the abuse spectrum.

Lots more.

Investing in the ecosystem

And we’ve built new tools to detect those shades of gray. I won’t divulge our entire budget for the Omnivore program, but I can tell you that we’re investing $20,000 per month on monitoring just one of those new “shades of gray.” Not to mention our investment in new staff, and in training. We are committed to protecting the email ecosystem. That’s not to say that our colleagues, like ConstantContact, ExactTarget, and other major ESPs, are not. They all devote a tremendous  amount of time, energy, money and resources on this stuff, and we’d be remiss not to mention them. Especially since they’re so willing to share their research with each other. Without a properly functioning ecosystem, we’re all dead.

We just want our customers to know how much thought goes into abuse prevention. It’s important to convey that.

For example, if we catch a spammer trying to hack away at our system, we almost always trace them back to some small, free ISP that they’re using to host all their malware. What do those ISPs usually tell us? “Abuse is inevitable and a part of life, and we’re sorry, and the account’s been shut down. Goodbye.” Hmm. We can’t help but wonder if they’re doing much to prevent that abuse in the first place. We’re sure they are, and we’re sure they’re being terse for legal reasons. But we still wonder.

We don’t want our customers to wonder.

New “Three Strikes” Policy

We don’t think Omnivore is perfect. But we’re much more confident in its ability to detect and prevent abuse now. So we’ll soon be implementing a new policy. If any sender on MailChimp is suspended three times in 6 months (whether the suspensions are a result of bad behavior or innocent mistakes, and whether the suspension was initiated by Omnivore or staff), Omnivore will permanently shut down the account. As I explained above, suspension isn’t always because of evil. Often, the sender made a totally innocent mistake. And after each suspension, our team always sends helpful recommendations to get senders back on the right track. We’ll even point some of them to 3rd party deliverability specialists, who can train them on best practices. So there’s rarely a valid reason for having 3 suspensions inside a 6-month period.

Related:

• Project Omnivore Declassified
• Spam lawsuits – What’s the worst that can happen?
• Preventing False Abuse Complaints