Here’s a nice email we got today:

“We just wanted to drop you a note to thank you for your patience and your advice last month as we migrated to your system.  We’re fairly confident now in our assumption that your fantastic deliverability was the cause for the elevated abuse complaint rate that we hadn’t seen before.  [Competing ESP] just didn’t get emails to those people it seems!

We had a few complaints this month as we migrated more of our database into your system, but they were at a manageable level and we saw complaints drop precipitously in the existing lists that had issues last month.  This is what we had hoped for and assumed would happen.  As our intake of new subscribers was at the normal rate, if the cause of our problems had been subscribers who felt duped somehow we assume the complaint rate would have stayed the same.  We’re very glad to see confirmed what we had assumed and hoped was the case.

So, this is just a note to apologize again for the problems last month and to say how much we appreciate your great service and your fantastic product.  We look forward to a long and fruitful relationship and we’ll always be here to answer to you if you have any questions or concerns   Thanks!”

The first paragraph is a common (and ironic) issue. Sometimes, better deliverability can actually get you into trouble. Some people send emails from their own servers for years, and don’t realize how bad their deliverability is. When they switch to an ESP, their deliverability improves, and suddenly their subscribers are getting their emails for the first time. What’s their reaction? Usually, something like, “Who are you? I signed up 2 years ago, and you’re just now contacting me with a newsletter out of nowhere?!?”

For the record, I have no idea if that sender will be fine moving forward. It’s very possible things will go wrong again, which is why we have automated processes constantly watching. But it’s always nice to see people who care about email, and want to take responsibility for what they’re doing.

Other encouraging notes:

“It is greatly reassuring to have both a great service and to feel this kind of security is in place to ensure deliverability.  Thanks a whole lot and I will be keeping watch to make sure it will not happen again!”

and here’s a bittersweet one from a user we unfortunately couldn’t help:

I wanted to thank you and the rest of the team there at MailChimp for the help on our accounts over the past few months.  We appreciate your efforts and your level of professionalism.  Job well done.

Unfortunately, the shareholders have expressed an interest in keeping the [REDACTED] and [REDACTED] brands paired and have secured another ESP for us to use.

Thanks, hope to work with you again and the [REDACTED] division will be staying with the Chimperoo.
: )

It’s been a looong time since I was running our abuse desk, but I can guess that the problem above was that the client wanted to combine two lists that were obtained by two different brands under two different permission circumstances. Our advice is to email each list separately to notify them of the impending “merger,” then move forward with the combined lists. Sometimes, people just don’t want to do that. It’s their prerogative, but it’s our servers. It’s nice when customers get that.

If you work at an abuse desk just stop reading now, knowing that there are indeed good people out there who care. Happy thoughts. Rainbows and unicorns!

If you don’t work at an abuse desk, and you’re wondering what the big deal is, read on.

Some Bad Letters. Sigh.

Now for the ugly stuff.

Since we’re a self-serve ESP with over 650,000 users sending more than 1 billion messages per month, we kinda have to put autonomous abuse prevention processes in place. We call one of our systems “Omnivore,” and it’s constantly scanning accounts and looking for warning signs like high abuse complaint rates, higher than average bounce rates, or bad email addresses on the list.  If it finds something suspicious, we temporarily suspend the account and send a few questions to the sender. We spend a lot of time making sure those questions we send aren’t accusatory, and we’re always recalibrating those messages.

Despite all the attention we put into crafting the most perfectly-balanced message, the questions still manage to get responses like this:

Bastards,
You had our account closed since feb 24th. For your slow review. I want my leads back and I will charge back an entire account for your terrible service. I have not done anything wrong. I gathered leads at the tradeshow from people I spoke with and they specifically asked to sent them PDF file with pricing information. Suspending the account for 20 emails accounts, you are out of your mind you stupid mailchimp company. Get a fucking phone support you bastards, so I can chew you up.

For the record, when people complain about “slow review,” 99% of the time it’s because we sent them questions via email that they refuse to answer. They always insist that they call us on the phone, to “explain their business model” or, like in this case, to “chew us out.” Sigh. There was a time when my co-founder and I handled this stuff (and these kind of people) ourselves. And we did it on the phone. That’s why we refuse to subject our staff to this. You either care about email etiquette, or you don’t. We don’t judge you if you don’t. You just can’t use MailChimp.

Here’s a follow up from that same user:

Seriously,

What a useless company. We have been paying you for so many months now. Spent hours learning your system and creating a tiny 25 email campaign and now we have to go back to Outlook. What is the point of your company then if we cannot have lists of our clients from the tradeshows.  Get a phone line! Talk to your customers and don’t simply hide behind chat and emails. Unprofessional company. I wish our cart would support [COMPETING ESP WITH NOTORIOUSLY BAD REPUTATION], cause you I’m very upset and I will looking somewhere else to spend money on emails.

Go and review our own links, I do not have time to look at your poorly written general tutorials. You cost me 4-6 hours of work today and I will not forgive that.

If I met 25 people at a trade show, who actually wanted my PDF (and weren’t just faking interest to make me go away), I’d email them personal, individualized emails. And yes, I’d do it from Outlook. That’s called sales. And in those emails I send — and in my PDF — I’d include links to more awesome stuff they could subscribe to (which is when a system like MailChimp comes into play).  Wait, let me back track. If I met 25 people at a trade show, who wanted my PDF, I’d be carrying the PDF. Already printed. Then I’d hand it to them. Or I’d email it to them right there, from my iPhone. Or I’d ask them to use a cool opt-in tool like — I don’t know – Chimpadeedoo. Point is, trade shows are fine and dandy. We’ve even built mobile apps for them. Just collect emails responsibly. But this guy’s a busy professional. No time to read tutorials. Kinda stings that he called our resources “poorly written” though.

Then there are the tweets.

We advised one user, who setup a list and got a very angry direct spam complaint, to contact his list and make sure they really wanted to hear from him (aka “re-confirm your list”). We know how it goes. Sometimes you collect a list of contacts that you’ve met over the years, and you want to get back in touch with them. No prob. But you should send them a nice email asking if they’re still around, and if they a) remember you, and b) still want to hear from you. If you send them an email marketing promotion out of the blue, you get spam complaints. You should re-confirm their permission. It’s the right thing to do.

We sent that advice to someone, and got this response:

Sorry, but your suggested process is unacceptable. I would rather consider it spam to receive a message to opt-in again to a newsletter that I subscribed to. This leaves me with the only option to move to another e-mail provider. I’ll inform my network about that.

and inform his network he did, with this tweet:

Mailchimp asking me to spam my complete network. Huhhh? Something is wrong here. #mailchimp #email

Hmm, we actually told him how not to spam his network, but I guess something got lost in translation.

Then there are the legal letters.

We recently received a very concerning letter from a company who told us they suspected some sort of data breach on their account, because their list was evidently being used by multiple competitors. We used our Email Genome Project servers to search our system, and found at least three other users — in the same industry and general geographic area — that had email lists with a high (greater than 90%) correlation with that sender. All were already shut down by Omnivore. In fact, the company who sent us the letter was just recently suspended from MailChimp too. It seems they were all using the same list (they most likely purchased it from the same source, or there was an insider selling the list to competitors). Whatever the case, it was a bad list that got consistent complaints, so we had to keep it off our system.

There was another legal letter from someone who got spam from a PR agency (read about how risky this industry can be for ESPs), who thought it would be a good idea to scrape emails from websites and blast out some client news. Unfortunately, one of the recipients was a guy who enjoys taking people to small claims court for violations of CAN-SPAM and other local state spam laws, in his pastime (he has a whole website boasting how many spammers he’s bagged). This, btw, is why we have an in-house counsel and privacy officer.

There’s some seriously nasty stuff going on out there, and the Compliance Team is here to protect our infrastructure against that nasty stuff. I’m not even going to talk about the money and resources we spend defending ourselves against users who get their computers infected by malware, and whose MailChimp accounts (not to mention all their other bank, personal, and business accounts) end up on the black hat market. People think we’re here to teach them how to run their business, or to “bust their balls for a couple spam complaints.” They don’t realize the level of threats the Compliance Team deals with on a daily basis, how brittle deliverability is, and what kind of work we go through to make sure our infrastructure has a good reputation.

If you know someone who works at an abuse desk somewhere, go ahead and give them a hug today.

If you’re interested in anti-spam topics, and you enjoy reading long technical documents composed in the courier font inside very tight margins (like I do), be sure to read this article about a best practices document that some blocklist operators have been working on since 2004:

“In any case, for those who choose to slog through it, BCP 07 offers insight into the way the major blacklist operators think”

I don’t handle our abuse or deliverability stuff anymore at MailChimp (much smarter people have taken over) but I remember the early days, when the behavior of just one MailChimp user would get our entire IP range blocked somewhere, and we’d have to jump into public forums and beg forgiveness (then endure all the ridicule) before getting delisted. It’s matured so much since then.

I also remember dealing with blocklists that banned MailChimp, but they were mysteriously also blocking the entire Internet too. So I found this interesting:

“A number of DNSBLs have shut down operations in such a way as to list the entire Internet, sometimes without warning.  These were usually done this way to force DNSBL users (mail administrators) to adjust their DNSBL client configurations to omit the now inoperative DNSBL and to shed the DNS query load from the registered domain name servers for the DNSBL. “

There are also guidelines in the document for DNSBL users, which make my imagination run wild about what incidents prompted the need for them.

This one, for example:

The DNSBL user MUST ensure that they understand the intended use of the DNSBL.  For example, some IP address-based DNSBLs are appropriate only for assessment of the peer IP address of the machine connecting to the DNSBL user’s mail server, and not other IP addresses appearing in an email (such as header Received lines or web links), or IRCconnections etc.  While a DNSBL user may choose to ignore the intent of the DNSBL, they SHOULD implement any variance in compliance with the DNSBL usage instructions.

For example, one of the requirements of some DNSBLs is that if the DNSBL is used contrary to the usage instructions, then the DNSBL user should not identify the DNSBL being used.  Furthermore, it is the DNSBL user’s responsibility to mitigate the effect of the listing locally.

Any system manager that uses DNSBLs is entrusting part of his or her server management to the parties that run the lists.  A DNSBL manager that decided to list 0/0 (which has actually happened) could cause every server that uses the DNSBL to reject all mail.

What I find interesting is not so much how someone at MailChimp put my face on one of the dancing elves (though I did get a chuckle out of my breakdance routine), but that this year, there’s a “Report for Abuse” link in the upper left corner.

I don’t recall that being on last year’s Elf thing.

This reminds me. While doing research for a presentation on “The Abuse of Freemium” early this year, I was amazed at how many places I found “report abuse” links. I put together this slide:

The point I was trying to make was that if you’re a startup that’s building a web app, you’re probably not thinking all that much about abuse in the early stages. You know it’s inevitable, but your attitude is probably, “we’ll cross that bridge when we get there.”

Rightfully so. You can’t let fear get the best of you.

But you will cross that bridge.

By the way, this is what the bridge actually looks like:

You’ll probably need to invest in something slightly more sophisticated than a machete though.

The experiment only involved sending roughly 7,000 email campaigns over to be reviewed. But within the first 2 days, we started getting back some unexpected, yet fascinating results.

In particular, there were certain email templates that kept getting repeatedly flagged as spam by these human reviewers, even though they weren’t spam at all.

All these “false positives” had some common design traits, so we thought we should share our findings…

How Did The Experiment Work?

When Omnivore detected an email that had traits of potential abuse, we sent it to Mechanical Turk. A copy of the email (sans private data, like recipient information) was displayed inside of an interface that looked something like this:

In general, we listed some rules at the top, then presented the campaign below it, then asked the reviewer to tell us if the email violated any of the listed rules, back at the top of the page. User Interface snobs will notice that in general, this interface looks like it was QWERTY-fied (designed to slow users down a little). We could’ve used very simple “Is this spam? Yes/No” buttons, but you don’t want people judging too fast.

How Effective Was The Experiment?

The experiment went as well as you’d expect, using people who weren’t heavily trained on the intricacies of permission-based email marketing. Generally speaking, Turkers like to work fast, so they’re best for picking out the most egregious offenders (think along the lines of porno or pharma spam). To that end, they’re great at catching the really evil spammers who try to penetrate into our system and send extremely bad stuff that would jeopardize our deliverability.

But when it came to reviewing an email from, say, a reputable business that purchased a not-so-reputable list from a local chamber of commerce, the reviewers experienced some difficulty. So crowdsourcing is good, but not a silver bullet with respect to abuse prevention (we are still crowdsourcing, but the experiment has changed significantly).

Though we weren’t thrilled with the initial results, this exercise revealed a lot about how people look at email design.

21 Seconds To Decide

Mechanical Turk measures how much time people spend performing each review, so we can tell when people are just clicking random stuff and moving on to their next task. On average, the human reviewers spent only 21 seconds reviewing these “false positive” emails. Now, we can’t read their minds, so there’s no reliable way of telling if they bothered to check for “permission reminders” or “CAN-SPAM compliance” in the footers. But it’s safe to say they weren’t doing a very thorough analysis. I’d wager that most of that 21 seconds was spent reading the criteria at the top of the interface, and not the email itself. They definitely weren’t visiting the senders’ websites to see if there was a proper signup form, and testing to see if they used opt-in best practices. They were making relatively quick, gut-level decisions on whether or not an email “looked spammy.”

The False Positives

Below are some email designs that kept getting marked as spam by Mechanical Turk reviewers. Keep in mind that at the time of this experiment, none of the senders of these emails were determined to be abusive. Their email stats suggested they were sending permission-based emails. Their recipients probably knew the emails were legit — but our independent reviewers did not.

1. Want to learn Photoshop?

In general, I think the above email has got some layout issues that make it look a bit sloppy. Their images are breaking the template. At the top, where people are accustomed to seeing a logo, the sender only used text. In fact, the text isn’t even the company’s name, but a bright red “salesy” kind of question: “Want to learn Japanese or Chinese?” Doesn’t exactly inspire confidence that you know your recipient, or what he’s interested in. Unfortunately, the Chinese characters don’t help their reputation much either. We’ve all received a bit too much of this in our inbox:

2. The Red Flyer

I’m sure that loyal customers of this local pizzeria were happy to get an offer for a free t-shirt:

But I don’t think our human reviewers liked the “hyperlink blue” verdana font, then the giant red “FREE” text below that (then the green text below that, then the blue text below that, then the gray text below that). Something about this email made it look more like a stock template for a flyer, not an email newsletter to loyal customers. I couldn’t help but think that the scrunched up airplane logo looked like those images that spammers try to skew, in order to get around anti-spam filters who scan the content of images:

Aside from the image quality issues, some extra copy could’ve been added to demonstrate that this email was being sent to their customers. Don’t get me wrong. T-shirt giveaways can be  extremely effective (here are some stats to prove it), but you should probably do more than just yell “FREE T-SHIRT!”

At the very least, an image of the actual t-shirt seems in order.

Here’s a nice example from ScoutMob:

3. Not Plain Enough Text

This email repeatedly got marked as spam by our reviewers:

You’ll notice it has no images. No branding, no logos, no photos.

Yes, one could make the case that plain, old-fashioned, text-only emails can be more personal, and therefore more effective under some circumstances.

But if you’re gonna go all-text, you need to go all the way, baby. Centered text, colored backgrounds, and colored borders look like you’re going for an HTML email look. But when you fail to include any logos or images, it looks half-baked. Like a spammer, getting all “Rich Text:”

Even if you don’t have a logo, one way of showing your brand is to include your website’s domain. But this sender used the bit.ly URL shortener instead:

In their defense, that’s probably because the link to the event they’re promoting was really long or something (webinar links get that way sometimes). The problem is that spammers are known to hide malicious links behind reputable URL shorteners (see: URL Shorteners and Blacklists), so that helpful little link just ends up hurting them.

4. Read it and Weep

This one was actually surprising to me, because I thought it was well designed:

The title font even looks customized (it’s not arial, it’s not verdana, and it’s certainly not comic sans). It’s laid out pretty nicely. The pink is a custom color, too. The only possible problem that I can see is that it’s extremely text-heavy, with zero images. To the untrained eye, it almost falls into that “not plain-enough text” category above, but this doesn’t look half-baked or sloppy at all to me. This email shows signs of actual craftsmanship and skill with typography (web design is 95% typography, right?). This sender’s subscribers are probably fine with all this text (the sender is an author, after all). But to our independent reviewers, this email apparently looked pretty spammy. In this case, I personally wouldn’t change my design or behavior. If I had to make recommendations, I’d consider adding elements that made it look more “newslettery.” Perhaps a small avatar of the author could be worked into the template’s footer, or some “share this on social sites” icons. If this is all about the written word, and images are forbidden, text can be ornamental too.

5. Set it and Forgot it

Senders that used one of our stock RSS-to-email templates seemed to get flagged the most:

As I write this article, we’re actually working on tweaking this template so that the header is more customizable (forcing the title to be ALL CAPS, in retrospect, was not a great idea).

But many of the bloggers who used this template didn’t bother customizing the RSS merge tags any further to include images from their posts. They didn’t customize the fonts, link colors, or anything at all, it seems.

I also wonder if, in some cases, the Table of Contents was so large, our independent reviewers didn’t bother scrolling down to look for real content. All they saw was a bunch of nonsensical looking TOC links. This happens if you update your blog frequently, but you schedule your RSS-to-email campaign to go out in weekly or monthly batches. Not that I’d change my behavior just for random Mechanical Turk reviewers. What your subscribers want is more important.

But there’s a broader lesson here on image vs. text balance. A similar example plucked from my spam folder in Gmail:

Why this is important to email marketers

When you send a lot of email marketing, even to a totally permission-based double opt-in list, you’re going to get some spam complaints from your recipients. It’s inevitable. Sometimes, it’s because they’re too lazy to click your unsub link, they think the “spam” button is the unsub link, or sometimes it’s because they forgot signing up to your list (maybe because you send infrequently, like me).

And sometimes, when your email is marked as spam, a human from an ISP, or a human from an anti-spam organization, will actually do a manual review of your email (See: “Who’s secretly reading your emails?”). Some anti-spam organizations use volunteers, who are driven by passion more than pay (nothing wrong with that, but you have to wonder how detailed their training is). We’ve experienced enough “your client’s email has been reviewed by our team, and determined to be spam, so we’re blocking your IP range” situations to know that those reviewers don’t always do a thorough analysis of your list management practices (not part of their job description anyway). This is partly why our own terms of use seem so strict to some. ISPs get complaints, they look at your email, and they make a split-second decision to “blacklist or not.”

So even if you do your list management right, and you design everything perfectly around your subscribers’ expectations, we always recommend that you give some consideration to this “secret” audience that also reads your email (See: “What makes a good permission reminder?”). Don’t bend over backwards for them, or anything.

It’s kind of like how your mother always told you to wear clean underwear, “in case you’re in an accident.” Take a good look at your email templates, and ask yourself, “If my email got reported as spam, and some spamcop laid his eyes on it, what would they think? Would mom be proud?”

Related:

• How your email design can get you blacklisted
• Stupid Email Design Mistakes
• How to avoid spam filters (the non-human kind)
• Want 700,000 HTML email templates? (more fun w/Mechanical Turk)
• Is your email marketing human?

For those of you who don’t know, we built Omnivore in order to prepare for our big Freemium plan that we launched back on September 1st, 2009. We didn’t want to offer a free email marketing service without having a scalable system in place to protect our deliverability (not to mention the sanity of our Compliance Team).  Good thing, too.

In just under a year, MailChimp grew from 85,000 users to over 430,000. We couldn’t have grown 5-fold like that without Omnivore.

Here’s an update on what we’ve learned so far…

Since September 1st 2009 Omnivore has:

• Issued 69,927 warnings to 24,119 users for exhibiting bad behavior. Warnings like, “Hey, we detected a lot of unsubscribes from that last campaign — if it continues, we’re going to have a deliverability problem.” Another warning example would be something like, “Whoah, that last campaign had a higher than normal bounce rate. Something’s going on with your list hygiene. Here are some tips for you to address that issue before it gets worse.” Warnings usually never need a reply. They’re simple observations that let you know something’s wrong, and if it’s not corrected, could lead to more issues, which could lead to suspension.
• Automatically suspended 8,770 users. This typically happens when Omnivore sees something really alarming, and just can’t allow an email to leave our system. When a user’s account gets suspended, it’s placed into a queue for human review. Our compliance team basically investigates to see if it was a false positive, sends tips to users if it was an innocent mistake, or in some cases, might decide to permanently shut down the sender.
• Of those suspended accounts, 1,879 ultimately had to be shut down. Shutdowns don’t always mean the sender was evil. Sometimes they just mean that a sender might be sending content that’s too risky, and receives more than the normal amount of delivery problems or abuse reports. Even if they’re totally innocent, they can still cause harm, and we have to shut them down.

When I first wrote about Omnivore, I was very careful to explain that it was new, and still had a lot to learn.

Over the last few months, it’s learned a lot.

New Shades of Gray

Our approach with Omnivore has been primarily to look for those things that spam filters don’t. For example, we could simply scan outgoing email with Spam Assassin’s criteria, and block offending messages. But that would only catch the “black and white” stuff. That’s fine for catching the horrible appendage-enlargement spam we’re all so familiar with. But ESPs deal with “ignorant spam” more than “evil spam.” Ignorant spam is harder to define. It’s a gray area.

Speaking of shades of gray:

And that’s the stuff Omnivore looks for.  Stuff that looks like perfectly legitimate business mail, and that would slip past most spam filters, but then generate a ton of spam complaints from recipients (traits that humans think are spammy, but that spam filters miss).

Since launching, we’ve discovered even more shades of gray in the abuse spectrum.

Lots more.

Investing in the ecosystem

And we’ve built new tools to detect those shades of gray. I won’t divulge our entire budget for the Omnivore program, but I can tell you that we’re investing $20,000 per month on monitoring just one of those new “shades of gray.” Not to mention our investment in new staff, and in training. We are committed to protecting the email ecosystem. That’s not to say that our colleagues, like ConstantContact, ExactTarget, and other major ESPs, are not. They all devote a tremendous  amount of time, energy, money and resources on this stuff, and we’d be remiss not to mention them. Especially since they’re so willing to share their research with each other. Without a properly functioning ecosystem, we’re all dead.

We just want our customers to know how much thought goes into abuse prevention. It’s important to convey that.

For example, if we catch a spammer trying to hack away at our system, we almost always trace them back to some small, free ISP that they’re using to host all their malware. What do those ISPs usually tell us? “Abuse is inevitable and a part of life, and we’re sorry, and the account’s been shut down. Goodbye.” Hmm. We can’t help but wonder if they’re doing much to prevent that abuse in the first place. We’re sure they are, and we’re sure they’re being terse for legal reasons. But we still wonder.

We don’t want our customers to wonder.

New “Three Strikes” Policy

We don’t think Omnivore is perfect. But we’re much more confident in its ability to detect and prevent abuse now. So we’ll soon be implementing a new policy. If any sender on MailChimp is suspended three times in 6 months (whether the suspensions are a result of bad behavior or innocent mistakes, and whether the suspension was initiated by Omnivore or staff), Omnivore will permanently shut down the account. As I explained above, suspension isn’t always because of evil. Often, the sender made a totally innocent mistake. And after each suspension, our team always sends helpful recommendations to get senders back on the right track. We’ll even point some of them to 3rd party deliverability specialists, who can train them on best practices. So there’s rarely a valid reason for having 3 suspensions inside a 6-month period.

Related:

• Project Omnivore Declassified
• Spam lawsuits – What’s the worst that can happen?
• Preventing False Abuse Complaints

Think you might need a refresher?

• Is My List Ok to Use in MailChimp?
• How To Grow Your Email List in 3 Easy Steps
• When Email Addresses Go Stale
• Warning Signs That Your Client is Spamming – Free Guide

so if you have role addresses on your account that you know should receive your email marketing, we make you manually input those addresses.

That’s because role addresses are built for functions, not people…

In other words, they’re often forwarded to multiple employees in a company, and they often change owners.  So it’s pretty obvious how sending your newsletter to a role address can lead to spam complaints that jeopardize the deliverability of our system. We even have a handy “tell me more” link to our knowledge base explaining all this, just in case. And manually inputting role addresses is a lot easier than manually begging to get off blacklists. Yet people still complain about having to manually input role addresses.

I think I know why. Even though we explain the reason for doing what we do, we offer no handy tips on how they can solve their problem quickly and easily.

So here’s a way you can deal with this. It’s not going to be super quick and easy, but if you really want those role addresses on your list, and if you really care about deliverability, it’s worth it.

After you import your list into MailChimp, and we provide you with a list of the rejected role addresses, download that list to your computer.

Next, go to the Lists page in MailChimp, and click on the “forms” link for that list you just created:

at the top of the next page, you’ll get a link to your MailChimp-hosted signup form:

Copy that URL.

Now go to your email program and BCC that link to the small handful of subscribers that you know are real human beings and that truly want your email marketing, but who insisted on using a role address when they originally subscribed to your list. I’m assuming it’s a handful. If it’s thousands of role addresses, (where BCC’ing is not a possibility), we’ve got a deeper problem here. Omnivore is likely to shut down your account, because a high percentage of role addresses pretty much smells like a purchased email list.

Anyway, send that handful of addresses a personal note from your own desktop email program, with your own email address as the reply-to, and using your own ISP or company mail server to distribute the message. If the prospect of dealing with the spam complaints and delivery issues that arise from mass-BCC’ing is bothering you now, then yeah—now you know why we feel the way we do about role addresses and preventing abuse complaints.

But if it’s just a handful of people you know, and who are already used to receiving emails from you, everything will be just fine.

Use a personal note like this:

Hello friends, customers, and subscribers.

I’m moving my email marketing to a product called MailChimp. Yeah, the name’s funny and all, but it’s actually a super powerful tool that will make my life a lot easier, and get useful content to you more efficiently and reliably.

Anyway, you signed up to my list a while back using your company’s role address. Something like “sales@” or “webmaster@”

Problem is, MailChimp won’t let me use that role address, because your company might be forwarding incoming mail to multiple people. Furthermore, those people will often change departments. It’s this philosophical thing they have. I dunno.

So this means that if you want to continue receiving my awesome content, please subscribe to my list using your own, individual  email address.

Here’s the link to sign up:

[link to your MailChimp signup form]

Regards,

__________

You might even consider customizing your welcome emails to include some kind of free prize, or free useful resource (like a whitepaper, PDF guide, whatever). That way, in the letter above, you can actually give people an incentive to go through “all that work” of signup up to your list again. Hopefully, your content’s so darn good, they’ll gladly go sign up, regardless of prize (but people still like those prizes!).

Again, we understand that this creates work for you, the publisher, and also work for that handful of recipients who signed up with role addresses. But over the years we’ve seen a lot of people get into a lot of trouble sending emails to role addresses that forward to someone who never signed up for anything.

It’s an unbelievable hassle proving your innocence to all the parties involved. You have to explain your situation to your ESP, the recipient who’s complaining, any ISP abuse desks that are blocking you, and that anti-spam organization that’s now globally blocking all emails that contain your company’s domain name (no matter where or who the emails are sent from).

Trust us. We know it’s an inconvenience, but when it comes to email marketing, an ounce of abuse prevention is worth a pound of role addresses.

In late 2008, MailChimp Labs began Project Omnivore. Our goal was to build a massively scalable tool for our abuse team that could predict bad behavior.

The experiment started with an nVidia Tesla supercomputer, then grew to a cluster of Amazon EC2 servers running a genetic optimization program for 2 weeks nonstop, running over 61 trillion email data comparisons.

This article shares some of the results of our experiment, and where the technology is taking us…

Why Is Omnivore Needed?

You know what the hardest part of running an Email Service Provider (ESP) is? Detecting ignorant spammers. They’re very different from evil spammers. See, it’s pretty easy to detect “evil” spam. You know, the pharmaceutical appendage enhancing stuff, phishing scams, and Nigerian prince (419) junk. Spam filters actually do a really good job of catching the evil stuff nowadays (not perfect, but pretty darn good, all things considered). And most ESPs employ some kind of spam filter (usually a variation of SpamAssassin) to scan outgoing emails in their queue. Either to prevent evil spam from tainting our reputation, or to “grade” the spamminess of a message.

But those spam filters aren’t designed to detect when an ignorant marketer doesn’t realize he’s spamming, and sends a mass email without permission (remember, the definition of spam is “unsolicited bulk email“).

Lack of permission, in an otherwise perfectly legitimate looking business email, is very subtle and much harder to detect.

I’m talking about when a well-meaning small business owner just wants to get the word out about his new store, and “blasts” an unsolicited email to a list he obtained from his local chamber or from a tradeshow. He didn’t mean harm, and he thinks he’s “just doing business,” but he’s actually spamming. While it’s a different flavor of spam, it’s still spam (again, see: definition of spam). This kind of spam is hard to detect because the content is often perfectly fine and doesn’t contain the normal keywords or traits that spam filters are trained to look for. But this flavor of spam can cost an ESP dearly, because they tend to generate the bad kind of engagement (high complaints, high bounces, high unsubs) that can get our IPs blacklisted by email gateways and ISPs.

How exactly does one detect the lack of permission in someone’s account? Across over 230,000 accounts? Sure, we’ve got a well-trained compliance team who can review a new user’s account, and in the blink of an eye, judge whether or not they’re going to cause trouble for us. But as good as we are, a human review team is just not scalable enough to deal with hundreds of thousands of senders. Not to mention that someone we might approve as a “good sender” can eventually become a “bad” sender. Rigorous, 24/7 account review becomes a necessity.

So our abuse desk decided long ago that we had to change the way we think about handling abuse. We began experimenting and analyzing massive amounts of data in 2008, which led to our list activity score feature. The idea here was to stop classifying customers as good or bad (and giving them access to special IP ranges for better deliverability), and start looking at their list management practices instead.

This then led to even more granular analysis: subscriber engagement tracking. We now treat email delivery differently, depending on the engagement level of your subscribers. Which is nice, considering ISPs are also looking at engagement to decide whose emails show up in the inbox or not. As a sender, you can segment your campaigns based on subscriber engagement, or clean out the inactive members.

But it was when we came up with the idea for our freemium plan that we knew we needed a completely automated, intelligent abuse detection system in place. Without a scalable abuse prevention system, there’d be no (scalable) way to protect the deliverability of our servers from the abuse that comes with free. So we stepped up our research and created Omnivore.

What Omnivore Does

Omnivore is a program that runs in the background and analyzes email campaign and user account data. Non-stop.

When it finds anything suspicious about a MailChimp user or his campaigns, it’ll do one of two things:

1. Send the user a warning for something that looks problematic.
2. Suspend a user’s account for something bad, send them a warning, and alert our abuse team to investigate the account.

What Omnivore Doesn’t Do

Most important of all, Omnivore doesn’t replace or reduce our human abuse desk team. And despite what some angry people out there might think (or tweet), Omnivore doesn’t shut down “totally innocent, opt-in users” with “absolutely no warning.” Humans review reports from Omnivore. If an account’s been suspended or flagged by Omnivore for problems, our team investigates. So long as the user is not obviously an evil spammer, we attempt to contact the sender with some advice or instructions for account reinstatement. If you’re curious about how our abuse team makes its decisions, check out these compliance tips.

How Omnivore Works

Chad, our lead engineer, headed up the Omnivore Project. I’ve asked him to provide some technical insight into how it all works.

Ben: Without revealing too much of the secret sauce, how does Omnivore work? I heard the team discussing something about “genetic optimization?”

Chad: Yes, in a nutshell, genetic optimization is a method of determining the best option from a large set of possible choices.  When the universe of possibilities is large enough, it isn’t practical to just try all of them and pick the best – you have to use an optimization algorithm to narrow down on the best choices.  Genetic optimization uses a process that roughly mirrors how natural selection processes can incrementally produce the fittest candidate over many generations, hence the name.  You create a population of possible options, then breed and mutate the top performers until you get a good enough solution to stop. Assuming that choices that are similar to each other will perform similarly, this can get you to a good answer relatively efficiently.

Ben: So how’d you apply that to email marketing and spam?

Chad: We took every bad campaign that had ever been shut down by our human reviewers as well as every bad campaign that managed to get through, and started looking for common patterns.  We know a lot about every campaign that goes through our systems, as well as every list we manage and customer we sign up.  Our human experts had a laundry list of the traits that scream “bad campaign”, but for this thing to scale we needed to be absolutely, mathematically certain.  So we used a series of large scale genetic optimization tests running against every campaign we’ve ever sent to confirm which traits were predictive, and how predictive they were.

We did this for both negative reactions (bounces, unsubscribes, abuse complaints) and signs of engagement (opens, clicks) to give our team a complete picture of the likely results of any campaign, before the campaign is ever sent.  If Omnivore sees something that it’s certain will be bad, it alerts the abuse desk to review the campaign before it’s let through the system.

Ben: I hear you tried this on the machines at the office and they were too slow?

Chad: Right – even early small-scale tests would run for weeks before giving good results. The full tests would have taken years to complete. We ended up getting an nVidia Tesla and writing the process in highly-optimized C code, which was able to give us our preliminary results in a couple of hours. After we knew our algorithm was pretty close to what we wanted, we converted the process to a giant Hadoop Map/Reduce program running on a cluster of Amazon EC2 servers for about 20 days to get the final results for the first version.  Smaller optimization processes still run continuously to test new ideas and refine the model.

Ben: So this is totally different than just checking all outgoing campaigns with a spam filter?

Chad: Yes. It’s using the detailed sender information that we have as an ESP to look for that permission “gray area” mentioned above.

More importantly, we needed to be sure that Omnivore would continue to be efficient and predictive as our customer base grew and morphed after the free program was put into place.  Unlike static rules or blacklist-based methods of detecting spam, all of the major Omnivore systems are learning algorithms that keep up with changing user behavior without losing their predictive power.

Ben: After all is said and done, any fun or surprising observations to share?

Chad: Some traits and keywords that we thought we should focus on were actually poor predictors of bad behavior. For example, highly-targeted campaigns don’t do much better than other campaigns when it comes to abuse or unsubscribe rates.  Other things that you’d think are totally irrelevant at first glance turned out to be effective predictors, like the length of the subject line.

Ben: So a subject line that’s too short, or um — too long — would be a sign of trouble?

Chad: Something like that. Keep in mind it takes a combination of traits that add up in order for Omnivore to determine “this looks like lack of permission.”

Ben: Any other interesting observations?

Chad: When we started this process, we went straight to our team of human reviewers to show us the patterns that they were looking at when evaluating a new customer.  A lot of it was right on the money – particular industries definitely have a profile, and the language used when describing where permission came from is crucially important. However, some of the patterns turned out to be less predictive, like having a mailing address displayed prominently in the content and some of the other details of CAN-SPAM compliance.  It was also a bit surprising to discover exactly how bad most spam filters are at predicting permission issues.  Whether or not a campaign passes any of the free or commercial spam filters generally has little impact on its predicted outcomes.

Results So Far

As MailChimp scales and sends more campaigns, Omnivore will collect more data and adapt. It’s by no means complete. There are switches and knobs we haven’t even turned on yet. We’re currently running some of Omnivore’s scanning in “observation mode,” and not letting it act on anything. As it gets smarter, we’ll gradually activate more functionality and grant it more decision-making power.

But so far, here are some of the results:

• As of January 6, 2010, Omnivore has automatically sent 19,581 warnings to 9,349 users for exhibiting bad behavior. Of course, we also include tips and pointers on how they can change their ways.
• Omnivore has automatically suspended 2,249 users since September 1st 2009.
• 861 of those users ultimately had to be shut down. We hate losing customers (because we love money), but no customer is worth jeopardizing the deliverability and reputation of our entire system.

Looking ahead (literally)

The reason we built Omnivore was because we wanted to change the way we think about abuse. The project involved so much data crunching that it resulted in some interesting byproducts. Our subject line suggester is one example, as well as the engagement ranking and segmenting tools we mentioned earlier.

But Omnivore is learning more every day, and is actually getting good at predicting not just bad behavior, but good behavior too. Here’s a snapshot from our internal dashboard:

As you can see, Omnivore’s predicting open and click rates for this particular campaign, along with the “bad” stuff. As we feed it more data, the margin of error narrows, making it a powerful new feature we could be offering to our customers one day.

Omnivore’s predictive reporting is changing the way we deal with abuse, but might end up changing the way we think about email marketing in general.

MailChimp’s abuse desk runs Cloudmark to perform occasional “customer audits.” We basically scan for problem campaigns on our system that might jeopardize the deliverability of our servers. What’s Cloudmark, why do we use it, and how does it work?

Cloudmark is an advanced “message security” system that protects more than 300 million inboxes and works with more than 100 of the world’s largest ISPs and mobile operator networks such as EarthLink, Comcast, Cablevision, Charter Communications, Cox Communications, NTT Communications, Sprint Nextel, Virgin Media and Swisscom, as well as hosted messaging providers, including domainFACTORY and NuVox.

So if you send lots of email marketing, it’s kind of important to know who they are.

But how does their spam filtering technology (its fingerprinting algorithm) work?

Well, it’s a secret. Understandably so, because if they told everyone how they work, that would kind of defeat the purpose.

But here’s what they will tell you (from their website sales material):

Cloudmark’s Advanced Message Fingerprinting™ algorithms were designed to target sophisticated spamming and virus proliferation techniques. Unlike rules, Cloudmark fingerprinting algorithms are extremely lightweight, each optimized to perform only minimal processing on a message. As a result, message throughput is extremely fast and less processing CPU is required.

And here’s how they explain their Fingerprinting algorithm:

So they’re taking chunks of your message (which I assume could be content, senderscore reputation, and code), and taking it out of the context of your email campaign. I don’t know if this is done for speed, or as some kind of “double blind” methodology or what. Then they classify the chunks into “fingerprints.” Then, they compare those fingerprints from your campaign with other fingerprints in their database that have been classified as spam.

This is where I invite any geek out there who knows way better than me to please comment below. Please.

What to do if Cloudmark blocks you

If you get blocked by Cloudmark (and our abuse desk sent you to this page), our recommendation is to take a long, hard look at your content. There’s something in there that looks spammy. Given that Cloudmark is installed across 300 million inboxes and +100 ISPs around the world, it’s safe to say that your campaign looks spammy to a LOT of people.

If you’re not sure what “looks spammy” means, I’m not so sure you’re ready to be sending lots of email marketing.

Okay, maybe that was a bit out of line. I work at the abuse desk, so I get jaded sometimes. So here are a couple resources you need to read quick:

• Most common spam filters triggered by MailChimp users
• How spam filters think, and how to avoid them

If you’re looking for a simple, silver bullet kind of answer for “how to just get me past the spam filters” prepare to be frustrated. There is no single answer. The best answer I’ve been able to tell people is:

1. Open up your email program’s junk folder.
2. Look at what spammers do.
3. Then, don’t do that.

Cloudmark is everywhere

We’re members of the ESPC, and once sat in on a presentation that Cloudmark gave to the group. It was fascinating. Mostly because it was a “marketing guy” talking, who actually knew his stuff. No offense to marketing guys or anything. He knew about this stuff, and in the cases where he didn’t, he was smart enough to admit it. I distinctly remember a slide in his presentation where he showed almost every single major ISP in North America using Cloudmark. IIRC, the only ISP not on the list was AOL. They’re even partnered with ReturnPath (who we’re also partnered with) so that they can pull in sender reputation data.

If you run an ESP (and manage the abuse desk at an ESP), it’s the kind of slide that makes you gulp really loud. So I’m really glad we’ve got this in place for our abuse desk. I’ll post something later about how we’re using it to make better decisions about email abuse, who we warn, who we suspend, and who we shut down.

The backstory is a MailChimp customer sent a campaign to an email list that they collected at an event a long, long time ago (Related: How old lists will kill your deliverability). One of their recipients forgot who the @#%& the sender was, and reported the email to SpamCop.

Forget about the whole issue of whether or not the sender is an innocent victim here, because their list was ‘opt-in.’

What really matters is the sender’s domain name could be tainted, and all their emails (no matter where they send from) could be blocked all over the globe.

Here’s how that happens.

See the “spamvertized web site” links in the screenshot?

Those are some of the domains that SpamCop found in the reported email.

The 3 domains that you see in the screenshot above belong to MailChimp.

The domains listed below them (that you can’t see) are domains that belong to the sender of the email campaign (I’m protecting their privacy here).

There are 3 ways we can get our domains de-listed from SpamCop:

1. Shut down the sender (the fastest way to get delisted)
2. Respond to this report, and provide documentation that proves the sender obtained opt-in permission from the recipient, so “as you can see, this is all probably a simple misunderstanding.”
3. It behooves me not to tell you the third way.

One way or another, MailChimp’s Abuse Desk will get our domains delisted from SpamCop. But if we find out that someone has intentionally violated our terms of use, how hard do you think we’ll try to get the sender’s domain names delisted?

If we find out the sender purchased an email list, or they had an old email list and thought MailChimp would be a convenient way to “clean it,” we’re not exactly going to go out of our way to help their domains get delisted from SpamCop as we show them out the door.

The point I’m trying to make is that anti-spam systems “remember” domain names that they find inside of reported spam.

So if we end up deciding to shut down this MailChimp customer with extreme prejudice, and they move to some other email service provider (ESP), their domain will still be remembered as an abuser by SpamCop (and probably other email gateways and firewalls around the globe too).

If you have bad email management practices, you can run, but you can’t hide from your own email reputation.

How do you prevent this from happening to your company’s reputation?

• Never send unwanted email
• Don’t surprise anybody with emails they wouldn’t expect
• Don’t assume that people on your list remember who you are
• Don’t send to old email addresses
• Collect proof of opt-in, just in case you’re reported to SpamCop. Without it, ESPs have little recourse but to shut down your account.
• In your emails, always include some kind of reminder as to how you got the recipient’s email address (you’re receiving this email because…”). Bare minimum, put that in your footer. If it’s your first email campaign, consider making it your first paragraph.