Oct 17, 2005

Spam Filters and Salmon (how spam filters think)

Spamfiltersandsalmon It’s a whole lot easier than you think for legitimate email marketing to be mistaken as "junk" by the spam filters. Spending countless hours writing, designing, and coding an HTML email newsletter, only to have it eaten up by a hungry spam filter, can be very frustrating. In order to prevent this from happening to your campaigns, you’ve got to understand how spam filters think.

So we’ve posted an article at the MailChimp website: "Spam Filters and Salmon" and we’re re-posting here on the blog.

You know those salmon you see on TV? They travel miles and miles from the ocean, fighting upstream against the current to spawn. They make that amazing, final jump up the waterfall, and gulp—they land in that hungry bear’s mouth. D’oh!

Bearvssalmon

That’s kinda like email newsletters and spam filters. You spend hours and hours designing, coding, and writing your email campaign. You test the HTML email on all the different email applications. You make it look just right. You finally deliver it to your opt-in list, and gulp—they’re eaten by spam filters.

It really sucks spending all that time on your email marketing, only to have it thrown away by your customers’ spam filters and never even opened. And it’s a lot easier than you think for a totally legitimate, fully opt-in email newsletter to get mistaken as spam (this is called a "false positive" by the spam filters). Take a look at your latest email campaign’s open rate. Is it dismally low (20%-30% is about average, depending on industry)? It could be that your emails are getting eaten.

How Do Spam Filters Think?

When you send your email campaign, all your little email messages have to get past hungry spam filters at your recipients’ ISPs, and then past hungry spam filters on your recipients’ computers. To keep your email newsletter from getting chewed up and spit into the junk folder, it helps to understand how spam filters think…


What Spam Filters See: What Spam Filters Think (in bear language):
Your Subject Line "Did the sender use "spammy" words and phrases? Chomp!"
Who the message is addressed "To:" "Did the sender even know the recipient’s name, or is it just addressed to an email (which they could have easily guessed, or programmed a script to generate)? Roar!"
The content of your message "Is HTML email coded properly? Did the sender take the time to create a plain-text alternative? Does the content contain spammy words or phrases? Grrr, Chomp, Mmmmm!"
Your IP address "Was the email sent from a server that is on a blackhole-list, or is known to be spam-friendly? Mmm, crunchy."
The sender "Is the sender friend, or foe? If the sender isn’t in my address book or contacts list, me eat now."
Your "from" email address "Is the email address faked? Is it from a free email account, or does it sound too anonymous? I better eat it, just to be safe."
Your domain name "Is the sender using a valid domain name that he owns, or is it an anonymous looking, or faked? Could be a spoofer. Me eat you now."
Advice from other spam filters "Has this same message (or sender) already been reported as spam by other recipients, or other spam filters out there? My friends were right—this IS good salmon."



Subject Line

The subject line is the easiest way to get your email filtered as spam. Even if your email doesn’t get filtered as spam, your subject line is how your recipients judge whether or not to open or delete your message, so it’s got to be relevant. Here are some tips for crafting your subject line:


  • Write it so that in the blink of an eye, a recipient knows: 1) who sent it, and 2) what it’s about. Also see: "Surviving the Inbox Whack-a-mole Game" at the MailChimp Blog.
  • Make it intriguing, but in a way that’s relevant to your audience, and that reinforces who sent the message, and what it’s about. For instance, if we sent a MailChimp newsletter with the subject line, "FREE WHITEPAPER INSIDE!" we’d be idiots. Something like, "Whitepaper: Advice for improving your newsletter open rates" would be better.
  • Avoid spammy words and phrases, like FREE, mortgage, insurance, act now, casino, limited time, coupons, click now, open immediately, etc.
  • Don’t!!! use !!! too!!! much!!! punctuation!!!!!
  • DON’T USE ALL CAPS! PEOPLE DON’T LIKE WHEN YOU YELL!!!
  • Avoid expletives (this one’s a no-brainer)


Who the message is addressed "To:"

You know when you check your mailbox (the old fashioned mailbox, not your email), and you get those direct mail pieces addressed to, "CURRENT RESIDENT" or "HEAD OF HOUSEHOLD"? That’s pretty impersonal, right? It’s painfully obvious that some computer generated that mail. Well, it’s the same with email. Don’t just address your email newsletter to "email address." Merge their first and last names into the to: field. Here’s how you personalize the To: field with MailChimp.


Content of Message

Spam filters look for "spammy" clues both on the surface, and "under the hood" of your email’s content:

  • Don’t use "spammy" words and phrases (see above) in your content. It’s easier than you think to accidentally use spammy words, like "New XXX-Large T-shirts Available" or "Join us for Casino Night!" or, "Click now to download." Put down your "Copywriting that SELLS!!!!" book when you’re writing your email newsletters. Many of the call-to-action "tricks" you might have learned in the direct-marketing world will just get you in trouble with spam filters.
  • Code your HTML email properly. If your HTML email is coded sloppily, you’ll look like a spammer. Broken images, missing tags, and non web-safe colors are some of the things they look for. Don’t use Microsoft Word to generate your code (the code it generates is atrocious). Heck, some spam filters will penalize you for using Microsoft Front Page to code your emails! Learn how to code HTML properly. Or pay someone to do it for you.
  • Don’t just send a giant graphic. That’s what spammers do. Your HTML email needs a "healthy balance" of graphics and text. If you’re just sending a simple invitation, or a simple promotional piece to your recipients, and all it takes is a simple graphic, you can still include text in the footer area, such as your "unsubscribe" link, your physical mailing address, etc.
  • Always include a plain-text alternative with your HTML email. Spammers are lazy, and don’t bother to do that. It may seem like a hassle to write both versions of your email, but you really need to do it. And don’t skimp on the plain-text version either. In the past, we’ve gotten a little lazy, and made our plain-text alternatives say, "You’re viewing this because you can’t view HTML email. Visit this URL to view our message in your browser." Nowadays, spam filters compare your plain-text alternative to your HTML email. If 95% of the message is in HTML, and 5% is in plain-text, that looks sloppy and lazy to the spam filters.
  • Check this link we posted on our blog to view the huge list of criteria that SpamAssassin uses to filter email.


IP Address

Some spam filters sync up with online "blackhole lists," which are published directories of spam-server IP addresses. If your email server’s IP address is on a blackhole list, the spam filters won’t let your email through. The bad news is that you can easily wind up on one of these blackhole lists, even if you’re not sending spam. For instance, if a spammer is using a server at your ISP, and his server is within your server’s "IP Range," all your emails could get blocked too. That’s like the post office blacklisting your "house," because you live in the same "neighborhood" as a known criminal. That seems like a drastic measure, but spam is a drastic problem.

It’s particularly a concern for email service providers (like MailChimp). If just one of our users does something stupid, they could get our servers blacklisted, and ruin it for all our users. This is why we’re so vigilant about watching all outgoing email from our queue.

What’s really scary is that anybody that runs an email server can create his own little blackhole list, and share it with others. It’s up to that server admin to determine who gets listed, why they get listed, and if/when it ever gets de-listed. Needless to say, under this scenario, "spite listings" can become a real problem.



The good news is that the major ISPs and email servers out there only sync up with responsible blackhole lists, not just any old blacklist it finds. SpamCop is an excellent example of a "responsible" blackhole list service, because it uses reports from a large network of registered users, and only blacklists servers that have surpassed a certain threshold. Furthermore, if spam reports for a blacklisted server subside, SpamCop proactively de-lists that server.


Sender’s Email Address: Friend or foe?

Some spam filters use the concept of, "never take candy from a stranger." If the recipient hasn’t "white listed" your email address as a "Friend" or "Buddy," or if your email address isn’t in their address book, contacts list, or "sent" folder, you’re perceived as a stranger. And if you’re a stranger, the spam filter will automatically categorize your email as junk, or it will use much more stringent criteria when judging your email’s "spaminess." When people sign up for your email list, it’s a good idea to ask them to "please place this email address in your address book, to ensure proper delivery…" Place that request on your subscription confirmation screen, and all welcome emails. Some people place that text at the top of every newsletter they send.

And while we’re on the topic of your email address, make sure you’re not using something anonymous, like a free email account (@yahoo.com, @hotmail.com, etc). You wouldn’t trust a company that couldn’t afford their own $30 domain name, would you? Setup a real email address with your company domain name. Get your users to whitelist it. Don’t change it.


Domain Name

Some spam filters will check to make sure that an email claiming to be originating from a domain name actually did originate from that domain name. This is called "authentication" and it’s slowly becoming more and more common. Emails that aren’t "authenticated" are either classified as "junk" or are flagged as "suspicious." At the time of this writing, authentication has not been widely adopted (due to the overhead it places on sending email servers, and due to varying standards from different vendors), but it’s definitely a promising technology to watch. To learn more, Yahoo! has published info on its own authentication method (DomainKeys). Microsoft has published details about its SenderID Framework here. Yet another emerging standard is the Sender Policy Framework, or SPF.


Community-based reporting

AOL uses community-based reporting to block unwanted email. The concept is pretty neat. Its millions of users get billions of emails every day. Whenever an AOL user clicks his "this is junk" button for a particular message he received, that "complaint" is sent to AOL’s server. If enough of its users report an email from you as junk, AOL will block all your future email to their servers. AOL’s not the only ISP to use community based spam filters, but they’re a great example. Here’s more information on AOL’s feedback loop.


Don’t Be Paranoid of Bears

Most of the sophisticated spam filters out there aren’t as "black or white" with their algorithms as you might think. They use a well-rounded combination of the above criteria. SpamAssassin is an excellent, example. It assigns certain "severity points" for each "rule" an incoming message breaks. For instance, using "CLICK HERE!!!" might get 0.5 points (for each occurence in the message), while using bright red fonts might get 0.1 points , and including the word, "V1AGRA" in the subject line might get 4.0 points. It tallies up the total score, and if it exceeds a certain "threshold" (which is set by the person who installed it on the server), the email is categorized as "spam." Many spam filters work the same way.

When you write your email newsletter, you shouldn’t be paranoid about using the phrase, "Click here" once or twice in the email. As long as the rest of your email is in proper order, and you don’t violate too many of the rules and criteria we’ve outlined above, spam filters won’t chomp your email down for dinner.