Dec 11, 2012

Delivery for Everyone, Even the Little Guys

Avid readers of our delivery-related posts might notice we put a lot of emphasis on a few major ISPs, like Gmail, Yahoo!, Hotmail, and AOL. With good reason, too—these four ISPs make up a little more than 50% of our total volume each month. Seriously, we send a lot of emails to them, so it would be easy to put the blinders on and focus only on delivering to these household names.

But let’s take a look at where the other 50% of our volume goes. In a recent week (considering our volume is in seasonal flux), we sent emails to 14.8 million different domains. Even if we remove the Big Four and all of their affiliate and international domains, we still sent to 14.8 million domains.

So how do 14.8 million domains squeeze into only 50% of our volume, almost equivalent to the same number of emails sent to the Big Four? Because we send fewer than 1,000 emails to 99.87% of those domains. Yep, only a teeny-weeny fraction of a percentage of domains actually receive more than 1,000 emails from us in a given week.

Understandably (at least we hope so), it would be difficult to monitor and research 14.8 million domains to the same extent as Gmail. But that isn’t to say we don’t care about them just as much. We love all of our domains equally, as if they were our little email babies.

When we hear about delivery issues to any domain, we put on our troubleshooting caps and get to work. Whether we send them millions of emails or only hundreds, we want to find the root cause of the issue and resolve it. Maybe it’s our inquisitive nature, our attention to perfection, or just wanting to do right by our users, but we’ll spend just as much time figuring out joe-bobs-barbeque-rib-shack-and-ice-skating-bonanza.co.uk.wow as hotmail.com. Luckily, Joe Bob is usually chomping at the bit to get his daily deals emails, so he’s willing to work with us to quickly solve the problem and get back to glazing hams while turning figure-eights.

For Instance?

Let’s take a look at some examples of common issues we encounter when sending to various domains, and some options we explore to get the email flowing again. By the way, we’re changing names to protect the innocent. We’re all in this together, after all.

Example 1: Online Store in the UK

A user contacted our support team concerning bounce-back messages received when sending emails to employees at an online retailer. Checking the bounce messages in their account, the sender was seeing “smtp;550 Too many messages from this connection,” which means we’re sending email too quickly.

Many small-to-mid-size companies don’t expect to be inundated with hundreds of emails from a single source. They might be running their email in-house or even—GASP!—at a shared hosting provider, so their thresholds for email can be set pretty low to avoid crashes and downtime. We can slow down the rate of delivery, but sending too slowly could delay emails for hours or even days if we send enough volume to a domain. The key is finding the right balance to ensure consistently successful delivery.

Throttling is the rate at which we send emails to specific domains. We have a global throttle rate in place so we don’t just blast everyone with billions of emails, but we also have specific throttle rates to speed up or slow down email delivery to specific domains. Because there are many variables in a throttle rate, including the number of concurrent connections, the number of recipients per connection, max messages per interval of time, yadda, yadda, yadda, testing can often take a few days as we wait for results and tweak as necessary.

Since we’re math-happy nerds who want to improve the situation as quickly as possible, we do a little figuring: An extreme slow-down of emails will make the bounces go away, but could delay delivery. Whipping out our trusty calculators, we determine how slowly we can send emails without significantly impacting send time, and voila! We implement the new rate, watch as bounces disappear, and then over the course of days and weeks, gradually speed up sending to a reasonable rate in case we see larger volume in the future.

In the case of the UK retailer, we started with a tried-and-tested throttle rate we’ve used for domains with a similar volume. Good news, everybody! It worked from the get-go, and left plenty of room for volume to climb.

Example 2: Meal Delivery Service

Another user hopped into a support chat asking why delivery to a particular domain, fewer than 1,000 subscribers, was taking a whopping 11 hours. Delivery was time-sensitive because the user sends lunch menus to employees at various companies they serve. Understandably, 11 hours is entirely too long to wait to find out what’s for lunch.

According to our logs, the receiving server was throttling us! We were getting connection errors after a certain number of emails, temporarily stopping more emails from going out. The emails would eventually deliver, but only a little at a time before the connection errors reappeared.

We ran some numbers and tried a throttle rate of five emails per minute. That’s really slow, but considering the low volume to this domain and the current send delays, we weren’t too worried about impact. Unfortunately, even five per minute was too much for this domain. While the delays dropped to six hours, we were still seeing connection errors.

We determined the domain had very strict filters to stop spam, and the delivery speed was never the core issue. Through the sender, we asked the recipient to whitelist our IP addresses. Whitelisting our IP addresses allows our emails to bypass spam filters and firewalls, which can often be very strict on corporate email servers.

Keeping a reasonable throttle rate, the emails delivered in under 45 minutes, and everyone knew what they were eating that day.

Example 3: Institute for Higher Learning in Texas

Our support team had been working on a case involving a university in Texas when they brought us in to take a look. Like many private organizations running internal email servers, schools tend to have very tough anti-spam policies. After all, students have enough distractions to keep them from studying, amiright?

In this case, a user was receiving bounces which looked something like this: “smtp;452 Too many recipients received this hour.”

The university is basically telling us we sent too many emails in an hour, and they were temporarily stopping any more emails from being received. Our system is smart enough to know a temporary bounce message when it sees one, and will re-send the email every so often for up to three days. However, if the block isn’t lifted, we eventually stop trying and soft bounce the subscriber.

When our go-to throttle rates didn’t seem to help, we did something wild and crazy—we found a contact at the university and picked up a phone. Now, you might be wondering, why don’t we do that in the beginning? Well, a lot of email receivers don’t publish their IT department’s contact information, especially if it’s a small or mid-size business. Also, they’re trying to stop spam, and revealing secrets about getting through spam filters can be counter-productive. It’s rare receivers will be so helpful, but the university, like Joe Bob’s barbeque, was anxious to receive important emails from this user.

We gleaned two critical pieces of knowledge from the university’s IT department. First, the secret throttle recipe, which improved delivery almost immediately. But most important was learning the university blocked IP addresses after receiving a certain number of emails with invalid recipients. Consider that this is a university where hundreds of students graduate every year, and their email addresses are removed from the system. It’s easy to see how some senders were hitting this limit and getting blocked, even if the recipient had originally opted in. With this knowledge, we were able to work with users to prune their lists and recommend students sign up with personal email accounts instead.

Success! Emails delivered quickly with only a 2.4% bounce rate—mostly from lingering, invalid email addresses.

What About This Domain Blocking My Emails?

If you’re having trouble delivering to specific domains, the best place to start is our support team. They’re trained to troubleshoot a lot of common problems. For instance, they can check delivery logs and see if a receiver accepted an email with a “250 OK” message. When emails disappear after a success message, it means the receiving server filtered the email further in, usually placing it into a spam folder or possibly a black hole. We recommend the receiver whitelist our IP addresses because, technically, the email was delivered.

On the other hand, when something strange is in the neighborhood, support knows who to call. Delivery Engineers: No domain is too small. We’re on the job and ready to believe you.