Before a campaign can go out, before it even hits the queues, our system has to build it.  Build what?  Doesn’t the user build the campaign?  Well, I guess I’d say the user designs the campaign. It’s our job to write all the headers, define all the merge tags, and package all kinds of related info into a huge data payload so your awesome content can get to its final destination.

The Payload

If your list has a hundred thousand subscribers, building that payload can be a significant task.  To keep things simple, we’ve always built the whole thing at once before sending it on.  That way, we could easily track whether a campaign had gone out or not.  Yes the payload was built and went out, or no it didn’t.  Unfortunately, for users with sizable lists, that meant waiting for the entire payload to build before sending the first email.  Booooo!

When users click the “send” button, they kind of expect the campaign to start sending.  After all, they didn’t click the “build my campaign” button.  That’s why we’ve started chunking the payloads.  For those with huge lists, we build multiple payloads in chunks of ten thousand.  Each chunk gets moving immediately, so your campaign can start going out sooner.  The process does involve extra checks and balances on our part, but the results are pretty cool. We’ve supercharged your large campaigns.

The Validation

Okay, so we’ve built the payload chunk, and everything looks good.  Of course, we’re not done yet.  It’s not enough to create the payload, we have to make sure the payload is appropriate.  I’m talking about really naughty stuff here, like non-unicode characters.

I believe I mentioned before that the payload is, among other things, a huge list of headers and merge labels.  I’m talking about the all the customized unsubscribe links, first names, time zones, and all kinds of stuff.  All this data has to be in the correct format, with the correct character encoding, and blah blah blah.  The number of ways to break a system are exhaustive, and the number of ways to send email without breaking anything are limited.

Let’s just say our developers have streamlined the myriad little checks and validations.  I’d go into detail, but the intricacies of PHP extensions are not a journey I want to explore right now.  In short, MailChimp is faster.  A LOT faster!

The Routing

Okay, all the merge tags line up, and all the headers are perfect, but the work’s not done yet.  We still have to get your campaign to the right IPs.  The router does the work of actually dividing the payload and sending all the various parts to the best possible starting point.

The upgrades we made here fall into the “simple but profound” category.  The Router grabs a payload chunk and splits it up.  Each chunk actually contains “hints” at where every subscriber should go.  For dedicated IP users, it’s their dedicated IP.  For shared IP users, it indicates the IP reputation that would best match the subscriber activity.  We call it a “hint” because the Router has the final word here.  If worse comes to worst and your dedicated IP isn’t available or that perfect IP is full, the Router can override the “hint” and still get your campaign moving.

To do all of that, the Router has to parse the payload.  Well, it turns out one of our Developers came up with a brilliant C extension to speed that up.  Yeah, it was that easy.

The Math

The Router also has to keep track of what’s been sent.  This is important because some ISPs impose hourly maximums.  That means, they don’t want to see more than x number of emails per IP per hour.  If we go over those rates, we’re basically throwing your emails away, and no one wants that.

In order to stay under these rates, the Router has to ping a database that keeps track of IP stats.  Even if this ping takes 1 millisecond, it would take almost 3 hours to do it ten million times.  Realistically, we were pinging the database a lot more than that, and even with parallel processing, it wastes time. One of our developers had a brilliant idea to to pull the data in batches.  What used to take almost 3 hours now takes about 10 seconds.  You’ve got to love that.

The truth is, we’re always looking for ways to make MailChimp more lovable. Sometimes that’s with shirts and hats, but more often it’s the peeps behind the scenes pushing for updates to speed, reliability, and whatever else they can think of.

So let’s take a look. The first two charts in our “Quick Insights” shows the email opens based on time of day and day in the week. The ‘y’ axis displays the percentage of opens on our tested data. This is an average per campaign rate and what the campaign can expect. It may look simple, but it’s very useful. (Scratching head) “So, what you’re telling me is Thursday around 3 pm is an awesome time to send my Morning Coffee Newsletter?” Not really. Keep in mind that you should use this data as a reference. You certainly don’t want to send your Morning Coffee Newsletter at 3 pm. That would be silly, but then again…some of us need a little afternoon pick me up.

(click on the images for a closer look)

Well, how can I really use this then? Say for instance you don’t send a Morning Coffee Newsletter and you normally send your campaign out at 10 pm. We encourage you to try something new and set up a Time Warp campaign and set it for example, at 3 pm. Or even create an A/B split campaign  based on your different campaign delivery times. See how it works for you. Ultimately, you want to send your campaigns where you see fit but, you may come to find that based on your industry, A/B split test results, or time warp campaign, that a different delivery time works better for your campaigns.

Next on the menu is “The Effects of Subscriber Recency on Open /Click Rates.” Recency is a fancy word for: occurring right before the present. So, when your subscribers sign up for a newsletter they are likely to be more engaged from the start. Again the ‘y’ axis presents the percentage of opens across our tested data. You might think its a bit odd to see a dwindling of engagement as time goes on however, dependent upon your industry and what type of campaigns you send, you can fine tune that engagement. If you start experiencing depleting engagement, it may be time to consider reengaging your subscribers.

Last but not least is the “Effects of Number of Links in Email on Click Rate.”
The ‘y’ axis presents the percentage of click rates across our tested data. As this graph shows, the more links you have, the more clicks you are likely to have. This is true, but remember it’s not always the case and could effect your engagement. We advise that the links are relevant and pertain to your campaign content. It is also important to minimize the use of shortened links and check for misspellings. If you use URL shortening, have a look here.  A healthy balance of links and general campaign content is always good. So don’t overload your subscribers with hundreds of links, as they may eventually stop opening emails in fear of a “total link assault.”

All in all, we want to provide you with rough benchmarks of our performance and what you can expect from your campaigns. Every data point should work symbiotic with each other to achieve optimal campaign performance. What you send matters to you and to MailChimp, which is why we created Chimp Charts. Now off to eat some banana bread!

That leads to a very interesting question… How are we doing?  After sifting through 500 or so IPs and crunching the numbers on over a quarter of a billion emails, I might have an answer.  There are graphs with colored lines and all kinds of explanations below, but within my data set one thing is true.  75% of the emails we receive are delivered within 5 minutes or less.  That’s pretty cool.

Begin Technical Babble

Before I throw the charts at you, it might help if we went over a few basics of how MailChimp delivery works.  When you send a campaign, your list is divided up and distributed amongst our shared IP pool.  Each IP has a FIFO queue for outbound emails, and the size of these queues directly affects your delivery time.

Clearly, we should divide your list evenly amongst all our IPs.  That would give us the fastest delivery time for sure.  Yep, it sure would.  It’d be really fast.  Are you getting the feeling we don’t do that?  Good, now let me explain why it isn’t the best solution.

At this stage, your campaign looks a lot like a highway.  One of the things we like to do is match your subscriber’s activity rating with the reputation of our IP.  We’re putting your best subscribers in the fast lane so they have the best possible chance for a successful delivery.  Of course, this means we can’t divide your list evenly.

If you’re wondering why we don’t send your campaign over all of our IPs, I have a simple answer.  “Hell is other people.”  I’m pretty sure Sartre would approve of me borrowing that phrase.

You see, every now and then we get a bad apple.  Sometimes it’s accidental and sometimes it’s malicious, but these rogue apples can get our IPs blocked.  By optimizing the number of IPs that any one campaign touches, we protect all of our users from the odd mushy apple.

More Complicated

This is where the highway analogy starts to get out of control.  Not only is your list divided among several IPs, but each IP has a separate connection for every receiving domain in its queue.  The receiving domain is the ISP your subscriber uses, like @gmail.com or @hotmail.com.  It’s worth taking a look at your list and noting which domains you send to the most.

The good news is that these connections can all send concurrently.  Of course, the ISPs themselves often throttle incoming email to their own preference.  You’ll notice in the charts below that one ISP in particular throttles heavily.

So it’s like there are different highways, but the lanes are kind of… No, it’s like each ISP is a different car manufacturer, and the speed limit is…  Okay, maybe if we all had flying cars and the toll booth was like a filter … Ugh, I honestly can’t think of a good way to picture this.  Feel free to make suggestions.

On top of all that mess, you’re not the only one sending a campaign.  Your emails are being queued along with everyone else who just hit the send button.  We do everything we can to minimize the queues, but there are certain times of day when our volume is off the chain.  It’ll make you think twice about the “schedule delivery” feature.

Actual Data

I took two weeks of data from our pool of shared IPs and added them up by hour.  The first chart shows the volume of emails we sent.  You can see our heaviest hours are between 9am and noon (EST), but from experience, that range shifts back an hour depending on the time of year.

The next few charts measure the time interval between when an email was queued and when it was accepted for delivery by the email provider.  Keep that last part in mind.  It takes two to deliver.  We send the email to the ISP, and they deliver it to your subscriber.  The first part I can measure.  The second part is anyone’s guess.

You can see that 85% of the emails queued between 10:00am and 10:59am were sent within 5 minutes of being queued.  For Yahoo, it took 45 minutes to deliver the next 10% (looking at the 95% graph).  To send the next 4.5% (looking at the 95.5% graph) of Yahoo emails took a whopping 285 minutes.  Yikes!

Notes

Why don’t you see a graph for 100%?  Well, it turns out that a lot of lists contain one or two addresses that don’t exist or can’t receive email anymore.  The way we figure it, there’s always a chance.  We’ll try to get the ISP to accept the email for up to three days, and sometimes it actually works.  It makes my charts look horrifying though, so I cut the numbers off at 99.5%.  Problem solved.

You may have noticed that Yahoo loves to throttle email.  If your list has a lot of Yahoo addresses, you’ll want to take this into account.  For those looking at the “All” line, it’s simply the sum of all the emails together.  As the percent completion goes up, the last few thousand emails tend to be heavily weighted with Yahoo addresses.  The “All” line reflects this by inching up as well.

I’m pretty happy with these delivery times. For most ISPs, 95% of the emails in our queues are sent out in 5 minutes or less. It says we’ve done a good job at balancing reputation and speed. In the second part of this series, I’ll go over all the stuff that happens to your campaign before it ever hits the queue. It’ll be a lot like going behind the scenes at Universal Studios. There will be flames shooting up out of nowhere and you’ll see that Freddie is an awesome robot.

Go to Delivery Speed, Part 2

There’s been a lot of good commentary on the scope and impact Facebook Messages will have on the email using community.  That’s a big community, by the way, and it happens to include my mom.  Bless her heart, she signed up for Facebook last week, and now I have to explain that email, chats, and text messages aren’t separate things anymore.  It was difficult enough explaining the difference to begin with! For our MailChimp users, I thought I’d go into a little more detail.

First and foremost, you need to know Facebook isn’t offering a robust email service.  They specialize in person-to-person communication, and it helps if both people are Facebook users.  If you take that away, the connection between email, chat, and SMS becomes less significant.  What you’re left with is a click heavy interface with an intentionally limited set of options that contains all of your communication behind a series of drop-down menus.  Whew!  So what does this mean for your subscribers?

Can they receive your campaigns?

Say an @facebook.com user signs up for your newsletter. The first thing you’ll do is send a confirmation email, but wait. If they don’t have “Everyone” set in their privacy settings, your confirmation will be dropped.  Yes, dropped.  That means it won’t show up in the inbox or the spam folder, but it won’t bounce either.  You and your subscriber are just out of luck.

This privacy setting can have a big effect on your open rates.

Okay, the privacy thing wasn’t an issue, and we got that first email through the door. At this point, they can change their privacy setting to “Friends Only,” and your campaigns will still get delivered.  Awesome! That’s great news for you and privacy geeks like myself, but I should pass on one warning.

Your emails are tracked and grouped by the reply-to address.  If you switch to a new address, you’re starting a whole new thread inside Facebook Messages.  That means new reply-to addresses are susceptible to privacy updates, so always be consistent. Otherwise, you might discover Facebook is dropping a lot of your emails.

All the emails you send are grouped into a single thread. These groups are separated by the "reply-to" address, but they appear under your "from" name.

Will they see your email?

Messages from bulk senders and other non-friends go to a special directory called “Other.”  As mentioned earlier, emails are grouped by the reply-to address with new emails appearing at the bottom.  When a subscriber clicks on your thread, they’re automatically taken to your latest email.  However, they’re looking at the plain-text version.  Clicking an “Expand” link brings up the html version, but we need to take a special look at these plain-text previews.

On the left is how your emails look after the subscriber enters your thread. They have to click the "Expand" link in order to see the HTML version on the right.

You can’t ignore the plain-text version of your campaigns.  There are certain content elements that cause Facebook Messages to cut the visible portion of your plain-text campaign.  Remember, we can’t count opens unless they expand the html version, so you’ll want to play around with this preview.   So what are the cut-off signals?

• Underscores (e.g. _HEADER_)
• Numbers separated by spaces (e.g. Sep 21, 2010 03:05 pm)
• I did a lot of testing, but that doesn’t mean I caught everything.  Please comment if you find something new!

By the way, these cut-off signals also apply to text-only campaigns.  Instead of an “Expand” link, there is a “Show Hidden Text” link that appears in a completely different spot on the preview.  Odd? Regardless, if you care about presentation, this is an important feature to keep in mind.

This side-by-side comparison shows the effect a date can have on the email preview. On the left, the date read "Jun 13, 2011 08:00 am." On the right, I removed the spaces to make it "Jun 12,201108:00 am."

Deliverability

Facebook’s policies toward bulk email senders can be summed up very simply: “Facebook only accepts bulk mail from highly reputable sources and does not offer support for any issues encountered.“  This may sound harsh, but it isn’t uncommon.  It means the best thing we can do to ensure delivery is to maintain our IP reputation.  That’s something we already do, and it’s why you’ll hear us talk again and again about permission based lists, double opt-in confirmation, and engagement.

The truth is, no one knows if Facebook Messages has a future or not.  Right now, MailChimp users don’t seem to have noticed.  We send 1.5 billion emails a month and see less than 10 thousand @facebook.com addresses in that time.  Of course, things could change at any moment.  Like a giant hovering its foot over your house, you really shouldn’t ignore Facebook.

Hi MailChimp,

My name is [NAME] and I am the Marketing Director of a Group Buying site in [COUNTRY] ([COMPANYNAME]).

I am currently looking to upgrade our Email Marketing System and am in conversations with [COMPETING ESP]. I have never used [COMPETING ESP] and have used MailChimp.

[COMPETING ESP] seem to think that they can gaurantee 20% better deliverability than MailChimp. Is that something you can disprove? How can you disprove this (e.g. comparison of technology, types of examples)

We will be sending many emails by the end of the year (in excess of 200 million). Can you guys effectively handle this type of volume?

This is the response that I proposed:

We published this page over here to show that so long as you, the sender, follow best practices, MailChimp’s infrastructure can help you achieve 99% (and sometimes 100%) delivery to the inbox. But if [COMPETING ESP] can actually get you 120% to the inbox, then wow. By all means, you should use them.

and that is why I am not on the front lines answering questions anymore. In fact, I’m not sure how I’m still allowed to answer comments on this blog.

Fortunately, our support team has a better, more professional response to this question (hyperlinks and emphasis below are added by me):

Hello [NAME],

Any email marketer can technically obtain 100% delivery so long as the following is in place:

• Data is collected through confirmable opt-in methods.  There are no assumptions about permission in play. (The account owners responsibility)
• Content sent is 100% spam filter safe. (The account owners responsibility, which we can help with this and this)
• Your delivery solution dynamically monitors and polices its environment for misuse and aggressively closes accounts that create situations that could harm other accounts within the system. (The ESP’s responsibility)

This is the technology that we use to police our service for bad apples:

http://www.mailchimp.com/omnivore/ (an overview)

http://blog.mailchimp.com/project-omnivore-declassified/ (more technical background)

You might want to review the account shut down stats for [COMPETING ESP] for insight into their protection measures.

This page provides additional deliverability information with regards to our service (including our definition of the word, and the seed list we use):

http://mailchimp.com/about/deliverability/

The delivery graphs there give examples of how someone that moves to our service, and who follow best practices outlined above, can quickly obtain 100% deliverability. There is a sample group buying site there, which you might find useful, as well as this article about daily senders, by someone on our delivery team.

Our system currently sends on average over 1 billion emails a month. So 200 million annually is not at all a problem.

I think, in a nutshell, the answer is that “Most well-established ESPs have gotten their infrastructure set up and proactively monitored to allow for extremely good delivery to the inbox. The rest is up to the sender. And if the sender does screw something up, it’s the ESPs responsibility to purge them from the system with extreme prejudice (whenever gentle hints and educational intervention are not enough).”

Other deliverability related stuff you may be interested in:

• How To Avoid Spam Filters (free guide)
• Email Delivery for Geeks (free guide)
• Warning Signs That Your Client is Spamming (free guide)
• Priority Inboxes and Measuring Engagement
• Daily Senders and Deliverability

Mario was super nice to give us his blessing for the outage:

We dedicated this song on blip.fm to Joe:

Hopefully, they’re not planning to wear matching shirts out together or anything:

Whenever your company sends transactional emails, does anyone watch for replies?

If not, you could be missing out on some good fun (and maybe losing a few precious Gmail engagement points).

The first few comments we got were understandably furious. But eventually, the conversation changed. We think we were on the way to a very constructive discussion. I really enjoyed the dialogue I had with people offline as a result of all this, and I want to thank all the email companies who commented here — CritSend, CampaignMonitor, PostMarkApp, and Al Iverson’s A-1 Super Awesome Home DSL Email Service.   I mean, don’t get me wrong. We’re competitors. We’re not going to be singing Kumbaya around the campfire with each other any time soon. It’s just nice talking to people who know their stuff. I wish the discussion could continue.

But my patience has been worn down. ReturnPath is naggi—asking me politely to take this post down, because they “don’t want to arbitrate arguments between their partners.” [I didn't realize we were asking them to arbitrate] I suggested that, as an independent, unbiased scoring system, they should just do what I do: ignore the bastards. Actually, I suggested the complainers needed to “grow some” and that ReturnPath ought to tell them so. But that’s not how ReturnPath rolls (thankfully, I guess).

Anyway, some of the arguments we heard about our posted methodology seemed to go like this: “Your methodology is flawed, because SenderScore penalizes IP addresses that send very low volumes, and that don’t have a high reputation. For example, I have an IP address that has GREAT inbox rates (um, trust me) but that have a low SenderScore.”

Well, yes. We know that low-volume, low-reputation IPs can get great deliverability.

Buuuut we happen to think that an ESP’s very job is to send high volumes of email while simultaneously maintaining a good IP reputation. We send out tons of email through our infrastructure, 24/7. If it were a race car engine, SenderScore’s our tachometer. Does it show actual vehicle speed? No. But it’s extremely indicative of engine performance. If we see our SenderScore drop from 94 to 70, there’s a problem with the engine. It’s time to pull over and get that deliverability fixed.

So to people who say “SenderScore is a bogus number” we respectfully disagree. It may not work for all senders, but it works for ESPs. The ones who send high volume. And want to measure their reputation.

Obviously, I am not concerned with what other ESPs think, or how they respond. But personally, I think that ReturnPath’s naggi — um, polite requests for me to pull this blog post is actually going to work against them. I think they’re defending the very people who are disputing the validity of SenderScore. On the one hand, that concerns me. On the other hand, I’ve always appreciated irony. And I absolutely hate the blinking red voicemail light on my office phone.

So this awesome blog post, which attempted to highlight the usefulness of ReturnPath’s SenderScore, is now officially yanked — at the request of ReturnPath.

Perhaps you’ve observed your bounce rate going up while your open and click rates went down.  Maybe you’ve had increasing problems with blocks, slow delivery, or a number of other issues.  As daily senders, you’re likely more aware of your deliverability than most, and it’s going to take a joint effort between you and your ESP to make sure it stays healthy.

The Facts of Life

You see, those first four thousand subscribers were in love with you.  They wanted your email every day, even if it was a text-only campaign with no colorful images sporting those eye-soothing rounded corners.  They never complained to their ISP about you.  If your email ended up in their spam folder, they dragged it out, added you to their address book, and sent a kind email to let you know there may be trouble.

That kind of devotion is rare, and it naturally diminishes as you grow.  The next group of subscribers love you, but they aren’t “in love” with you. Then you met a bunch of people who just like you.  I mean, they like you a lot, but they don’t like-like you the way your core subscribers do.  It’s okay, we understand.  If we can just re-inspire those original warm feelings, you’ll be seeing the love from your campaign stats in no time.

What You Can Do

I’m about to give you a secret sauce of magical solutions that will improve your campaigns in every way, but you won’t like it.  In fact, you’ll laugh at how naive and ridiculous some of these ideas are.  I’ll be so embarrassed!  For my sake, maybe you should just skip ahead.  No?  Okay, but I warned you.

Magical Solution #1

Send less email.  It’s simple really, only send your daily email to subscribers who want it every day.  Offer a weekly or monthly version to everyone else.  Your open and click rates will go up, and fewer people will feel compelled to hit the “Spam” button.

Pro Tip: You can target your most loyal and newest subscribers at the same time. When choosing a list for your next campaign, click “send to segment.” Then set the Member Rating and Date Added filters as in the picture below. If you’ve had a spat of complaints or hard bounces and omnivore had to send you a warning, list segmenting is the quickest way to rehabilitate your account.

Segmenting List Inside a Campaign

Magical Solution #2

Improve your content.  If you need to send to your entire list every day, then make sure the content is rich and well rounded enough to be compelling to all.  ISPs are creating tools that put more and more power in the hands of your subscribers.  Every unopened email could be hurting your chances of reaching the inbox next time.

If applicable, consider segmenting your list by content.  Instead of sending the same long email to everyone, send the tech portion to your tech people, the shopping portion to your shopping people, and so forth.

While preparing for this blog, I came across something Ben wrote a few months ago.  It caught my eye as a good example of how you can and should take a hard look at your click stats.  The people who are exclusively clicking your tech, italian food, outdoor equipment, or potty humor links are prime candidates for targeted campaigns.  Yes, I am giving you the green light on an all potty humor email… for the customers.  I certainly don’t find that kind of thing amusing.  I’m too mature.  What was your user name again?

Magical Solution #3

Revise your list collection strategy.  Make sure your list is double opt-in to avoid hitting any spam traps or fake email accounts.  Be very clear with everyone who signs up that they will be receiving a daily email.  If there is a newsletter checkbox when a subscriber creates an account on your site, call more attention to it.  Subscribers who aren’t expecting your email are more likely to mark it as spam.  If enough people do this, the ISP may block you entirely, and by “you” I mean us!

What We Can Do

I know, I know, I’ve asked you to make a lot of hard decisions, and you want to know what we can do to help shoulder the burden.  After all, isn’t that why you signed up with an ESP in the first place?

The answer to your question is yes, we can help.  Actually, all good ESPs are already doing a ton to make your deliverability great.  This is the part where I’m going to overwhelm you with technical jargon and not explain what any of it means because it would take too long: DKIM, Reverse DNS, MX Records, Engagement Monitoring, IP Reputation, Blacklist Monitoring, Bounce Handling, SPF, MTA, Registrar, Throttling, Spam Filters, VERP, FBL, Whitelisting, Certification, Static/Dynamic Error Handling, Email Headers, Seed List, etc

If you’re interested, all of those terms and processes are explained in our Email Delivery For IT Professionals guide.  For daily senders, here are a few subjects you may want to consider when talking to your ESP:

Dedicated IPs

More than likely, your ESP has a sizable pool of shared IPs that all their users send over.   At MailChimp, we actually have several pools of IPs separated by cleanliness.  That way, emails to your best subscribers go out over our cleanest IP pool, and we can isolate emails to unproven subscribers so they don’t interfere with the rest of your campaign.  However, high volume senders can benefit from leaving the shared pool and purchasing a dedicated IP.

I’ll discuss many benefits of a dedicated IP below, but they key is that we’re helping ISPs trust you.  As a daily sender, your volume can look aggressive and dangerous, and this raises flags no matter how good your content is.  By sending to an active list from a stable location, we can make that volume look more attractive.  If you’re trying to get your subscribers to fall in love with you again, think of the ISPs as parents. If you can get their approval, you’re a big step ahead of the game.

And uh, I guess that makes Freddie your wingman.  Yes!  Thank you, thank you.  I’ll be here all night.  Someone get me some water because this metaphor is on fire.

Ahem…

Custom DNS

So I mentioned that your ESP has a lot of IPs, and each one of those has a domain name.  The domain names probably sound innocent and dull, like email1.campaigns.com.  However, under ideal circumstances an ISP would like to see the domain name match up to the From Address in your campaign.  While most of our users have a great sending history without a custom DNS, a lot of high volume daily senders can benefit from this kind of attention to detail.

Bounce Alerts

Bounces are both undesirable and inevitable.  Even the best list collection policy can’t avoid getting a few problem email addresses every now and then. We like to know the general makeup of your lists so we can be more responsive when bounces do occur.  Each ISP has their own bounce syntax and requires their own unique response.  Knowing what to look for can really help us prevent small problems from becoming big problems.

Any good ESP will already be familiar with bounce messages from Yahoo, Gmail, Hotmail, AOL, Comcast, or any other big ISP.  Beyond that, location is a key factor.  If the bulk of your list is in South America, your European ESP may not be as familiar with the receiving ISPs.  That doesn’t mean you need to switch ESPs, but you should get in touch with them to make sure your issues are on their radar.

Throttling

If it were up to us, we would send your email as fast as computationally possible, but the ISPs would never go for that.  You see, they need time to sniff every email for the salty scent of spam or the sickly odor of viruses. If we send too much too quickly, they’ll assume all the email is bad and throw it away.  The very thought gives us stomach cramps, so we do everything we can to assure your emails are sending at the right speed.  For those on dedicated IPs, your throttle rate should be fine tuned to match your IP reputation.

ISPs are also fond of sending bounce messages saying, and I’m paraphrasing here, “Something smells odd, and we’re going to need time to investigate. Don’t send us any more email for an hour or I’m throwing it away and adding you to my list of enemies.”  Ignoring those warnings can be very unhealthy, so it’s important your ESP respond as quickly as possible when those bounce messages occur.

You’ve Come a Long Way

For any email marketer, it’s probably worth reviewing your sending strategy every time your list grows by an order of magnitude.  With daily senders, those growing pains are going to be accelerated.  Just think, with a list of thirty-four thousand subscribers you’ll send a million emails a month. That’s definitely cool to email geeks like us.  However, at some point sending out more email becomes less valuable than sending out better email in the right way for your subscribers.  Like any good wingman, we want to see you succeed so we can brag about how we were there when it happened.

The experiment only involved sending roughly 7,000 email campaigns over to be reviewed. But within the first 2 days, we started getting back some unexpected, yet fascinating results.

In particular, there were certain email templates that kept getting repeatedly flagged as spam by these human reviewers, even though they weren’t spam at all.

All these “false positives” had some common design traits, so we thought we should share our findings…

How Did The Experiment Work?

When Omnivore detected an email that had traits of potential abuse, we sent it to Mechanical Turk. A copy of the email (sans private data, like recipient information) was displayed inside of an interface that looked something like this:

In general, we listed some rules at the top, then presented the campaign below it, then asked the reviewer to tell us if the email violated any of the listed rules, back at the top of the page. User Interface snobs will notice that in general, this interface looks like it was QWERTY-fied (designed to slow users down a little). We could’ve used very simple “Is this spam? Yes/No” buttons, but you don’t want people judging too fast.

How Effective Was The Experiment?

The experiment went as well as you’d expect, using people who weren’t heavily trained on the intricacies of permission-based email marketing. Generally speaking, Turkers like to work fast, so they’re best for picking out the most egregious offenders (think along the lines of porno or pharma spam). To that end, they’re great at catching the really evil spammers who try to penetrate into our system and send extremely bad stuff that would jeopardize our deliverability.

But when it came to reviewing an email from, say, a reputable business that purchased a not-so-reputable list from a local chamber of commerce, the reviewers experienced some difficulty. So crowdsourcing is good, but not a silver bullet with respect to abuse prevention (we are still crowdsourcing, but the experiment has changed significantly).

Though we weren’t thrilled with the initial results, this exercise revealed a lot about how people look at email design.

21 Seconds To Decide

Mechanical Turk measures how much time people spend performing each review, so we can tell when people are just clicking random stuff and moving on to their next task. On average, the human reviewers spent only 21 seconds reviewing these “false positive” emails. Now, we can’t read their minds, so there’s no reliable way of telling if they bothered to check for “permission reminders” or “CAN-SPAM compliance” in the footers. But it’s safe to say they weren’t doing a very thorough analysis. I’d wager that most of that 21 seconds was spent reading the criteria at the top of the interface, and not the email itself. They definitely weren’t visiting the senders’ websites to see if there was a proper signup form, and testing to see if they used opt-in best practices. They were making relatively quick, gut-level decisions on whether or not an email “looked spammy.”

The False Positives

Below are some email designs that kept getting marked as spam by Mechanical Turk reviewers. Keep in mind that at the time of this experiment, none of the senders of these emails were determined to be abusive. Their email stats suggested they were sending permission-based emails. Their recipients probably knew the emails were legit — but our independent reviewers did not.

1. Want to learn Photoshop?

In general, I think the above email has got some layout issues that make it look a bit sloppy. Their images are breaking the template. At the top, where people are accustomed to seeing a logo, the sender only used text. In fact, the text isn’t even the company’s name, but a bright red “salesy” kind of question: “Want to learn Japanese or Chinese?” Doesn’t exactly inspire confidence that you know your recipient, or what he’s interested in. Unfortunately, the Chinese characters don’t help their reputation much either. We’ve all received a bit too much of this in our inbox:

2. The Red Flyer

I’m sure that loyal customers of this local pizzeria were happy to get an offer for a free t-shirt:

But I don’t think our human reviewers liked the “hyperlink blue” verdana font, then the giant red “FREE” text below that (then the green text below that, then the blue text below that, then the gray text below that). Something about this email made it look more like a stock template for a flyer, not an email newsletter to loyal customers. I couldn’t help but think that the scrunched up airplane logo looked like those images that spammers try to skew, in order to get around anti-spam filters who scan the content of images:

Aside from the image quality issues, some extra copy could’ve been added to demonstrate that this email was being sent to their customers. Don’t get me wrong. T-shirt giveaways can be  extremely effective (here are some stats to prove it), but you should probably do more than just yell “FREE T-SHIRT!”

At the very least, an image of the actual t-shirt seems in order.

Here’s a nice example from ScoutMob:

3. Not Plain Enough Text

This email repeatedly got marked as spam by our reviewers:

You’ll notice it has no images. No branding, no logos, no photos.

Yes, one could make the case that plain, old-fashioned, text-only emails can be more personal, and therefore more effective under some circumstances.

But if you’re gonna go all-text, you need to go all the way, baby. Centered text, colored backgrounds, and colored borders look like you’re going for an HTML email look. But when you fail to include any logos or images, it looks half-baked. Like a spammer, getting all “Rich Text:”

Even if you don’t have a logo, one way of showing your brand is to include your website’s domain. But this sender used the bit.ly URL shortener instead:

In their defense, that’s probably because the link to the event they’re promoting was really long or something (webinar links get that way sometimes). The problem is that spammers are known to hide malicious links behind reputable URL shorteners (see: URL Shorteners and Blacklists), so that helpful little link just ends up hurting them.

4. Read it and Weep

This one was actually surprising to me, because I thought it was well designed:

The title font even looks customized (it’s not arial, it’s not verdana, and it’s certainly not comic sans). It’s laid out pretty nicely. The pink is a custom color, too. The only possible problem that I can see is that it’s extremely text-heavy, with zero images. To the untrained eye, it almost falls into that “not plain-enough text” category above, but this doesn’t look half-baked or sloppy at all to me. This email shows signs of actual craftsmanship and skill with typography (web design is 95% typography, right?). This sender’s subscribers are probably fine with all this text (the sender is an author, after all). But to our independent reviewers, this email apparently looked pretty spammy. In this case, I personally wouldn’t change my design or behavior. If I had to make recommendations, I’d consider adding elements that made it look more “newslettery.” Perhaps a small avatar of the author could be worked into the template’s footer, or some “share this on social sites” icons. If this is all about the written word, and images are forbidden, text can be ornamental too.

5. Set it and Forgot it

Senders that used one of our stock RSS-to-email templates seemed to get flagged the most:

As I write this article, we’re actually working on tweaking this template so that the header is more customizable (forcing the title to be ALL CAPS, in retrospect, was not a great idea).

But many of the bloggers who used this template didn’t bother customizing the RSS merge tags any further to include images from their posts. They didn’t customize the fonts, link colors, or anything at all, it seems.

I also wonder if, in some cases, the Table of Contents was so large, our independent reviewers didn’t bother scrolling down to look for real content. All they saw was a bunch of nonsensical looking TOC links. This happens if you update your blog frequently, but you schedule your RSS-to-email campaign to go out in weekly or monthly batches. Not that I’d change my behavior just for random Mechanical Turk reviewers. What your subscribers want is more important.

But there’s a broader lesson here on image vs. text balance. A similar example plucked from my spam folder in Gmail:

Why this is important to email marketers

When you send a lot of email marketing, even to a totally permission-based double opt-in list, you’re going to get some spam complaints from your recipients. It’s inevitable. Sometimes, it’s because they’re too lazy to click your unsub link, they think the “spam” button is the unsub link, or sometimes it’s because they forgot signing up to your list (maybe because you send infrequently, like me).

And sometimes, when your email is marked as spam, a human from an ISP, or a human from an anti-spam organization, will actually do a manual review of your email (See: “Who’s secretly reading your emails?”). Some anti-spam organizations use volunteers, who are driven by passion more than pay (nothing wrong with that, but you have to wonder how detailed their training is). We’ve experienced enough “your client’s email has been reviewed by our team, and determined to be spam, so we’re blocking your IP range” situations to know that those reviewers don’t always do a thorough analysis of your list management practices (not part of their job description anyway). This is partly why our own terms of use seem so strict to some. ISPs get complaints, they look at your email, and they make a split-second decision to “blacklist or not.”

So even if you do your list management right, and you design everything perfectly around your subscribers’ expectations, we always recommend that you give some consideration to this “secret” audience that also reads your email (See: “What makes a good permission reminder?”). Don’t bend over backwards for them, or anything.

It’s kind of like how your mother always told you to wear clean underwear, “in case you’re in an accident.” Take a good look at your email templates, and ask yourself, “If my email got reported as spam, and some spamcop laid his eyes on it, what would they think? Would mom be proud?”

Related:

• How your email design can get you blacklisted
• Stupid Email Design Mistakes
• How to avoid spam filters (the non-human kind)
• Want 700,000 HTML email templates? (more fun w/Mechanical Turk)
• Is your email marketing human?

• On the Classification of Bounces over at Suremail’s blog.

Related: Email Delivery for IT Professionals, a free guide on “how to build your own MailChimp,” in our Resources area.