May 11, 2010

Reddit’s Outage

Someone pointed me to this blog post form Reddit about their recent outage. Mainly because I kinda follow the topic of outages.

I thought they handled it well, but that’s not why I’m writing. Those guys know what they’re doing. I’m writing because while reading through their technical details, I stumbled upon this snippet about email deliverability…

email verification irony
We’ve also discovered that a lot of the verification emails we’ve been sending out haven’t been going through. It seems that the mail server admins at some popular domains (e.g., comcast.net, rr.com, adelphia.net, and me.com) have their servers configured to consider all mail from reddit to be spam. This is because Trend Micro has marked Amazon’s entire EC2 network as a "dial-up pool", and the aforementioned domains subscribe to Trend Micro’s list and block all mail from anyone on said list. We’ve written to Trend Micro explaining that we’re actually neither a spammer nor an individual end user, but rather an honest website that’s kind of a big deal, and they sent us a form letter explaining how to configure Outlook Express and encouraging us to ask our ISP for further information. We’ll try to figure something out as soon as time allows.

Yep. Welcome to our hell. Email delivery (on a massive scale) is kinda complicated.

Should anyone else out there have to setup an email delivery server all by yourself, this very topic (cloud-based MTAs) is covered in our guide:

Email Delivery for IT Professionals on page 5, under Hosting and Hardware.

…And you don’t want your MTAs to live in the cloud or in a virtual  environment. MTAs need bare metal and fast drives. An MTA’s activity  is bound mostly to disk and CPU, so cloud and virtual environments  aren’t suited well to the task. Consider this when you’re thinking about  your network topology and designing your application. Your sending  infrastructure (or at least your MTA) may have to exist in another  datacenter. We’ve seen the effects of tying your sending infrastructure  in the cloud, and the results are poor performance and reputation.

Amazon EC2 is great for massive computing in the cloud. We’ve used it to run 61 trillion data comparisons for Omnivore. But it’s not-so-great for mass email marketing.

In case anyone out there thinks that was kinda preachy sounding of me, trust me: we only know this stuff because we’ve tried it already (not because we’re any smarter).

While on the topic of MailChimp’s free guides, I’d like to take this opportunity to point out our spiffy new MailChimp Resource Center. Look at all those redesigned covers! We hope you’ll agree it’s not your average whitepaper directory.