MailChimp’s HQ is in Atlanta, Georgia, but we manage app servers and email delivery servers in data centers that are spread out geographically, in order to minimize risk from natural disasters. Our servers are in Dallas (Texas), Seattle (Washington State), the District of Columbia (Virginia), and New York. The New York data centers have prepared admirably for Hurricane Sandy, but we’ve received word that one of them is literally underwater right now. Human safety is obviously a priority over things like servers and bits and bytes and newsletters, so to be honest, it feels almost trivial posting an update about that stuff. But businesses who rely on MailChimp need to know, so that they can prepare too.

Our customer-facing app servers (the ones that power MailChimp) are not in the path of this storm (some are in DC, but the brunt of the storm has moved away from there), so MailChimp functionality such as building newsletters, checking stats and managing lists should not be affected. However, a few of our email delivery servers are in New York. They’re all okay right now, but we’d like to set some expectations…

Early this morning, we received news that a handful of delivery servers went down around midnight, but then we got news that they’re back up. Then we got word that they never went down in the first place,  and are running on backup diesel power. So all we can say for sure is that the news from New York is still developing, and we’ll have to keep monitoring the situation closely.

All you need to know right now: Just don’t re-send

We’ve designed our infrastructure so that if a delivery server fails, your campaign will get redirected to the other available servers for delivery. Still, email delivery could slow down. If your email isn’t delivered as fast as normal, it could just mean our servers are under heavy load. So please don’t re-send your campaigns. Currently, all delivery servers are running fine, and we’re at a typical Tuesday morning workload.

We’ll post relevant updates to the bottom of this blog post, and on our status account on Twitter. You can also track server status here. And our support team can answer questions about your campaigns.

UPDATES:

• 10:14am – We’ve been told that the New York data center that holds a few of our delivery servers is running on diesel power, and has roughly 60 hrs remaining. Our delivery team is operating under the assumption that they only have 30 hrs, and plans to manually switchover to our other data centers in case these fail in the middle of the night. This will help us prevent lost emails.
• 11:22 – Aaaaand we just received notice that they’re almost out of fuel already. Just to be safe, we’ve manually shut down that server and diverted email to our other data centers.
• 5:28am Wednesday – Delivered 45 million emails during the storm, with no customer-facing downtime. Some of it due to our planning, a lot of it due to our data centers’ preparations, and most of it due to sheer luck. Still some cleanup to do with re-balancing delivery queues.
• 1:40pm Wednesday – We’re told one of the downed data centers will be safe to light back up in an hour or so. Until then, we’re still running fine, albeit with fewer delivery servers online. We’re about to hit a peak sending period, so we expect delivery to be slower than normal. Same advice: don’t re-send.
• Retroactive-update: Customers in Europe and the east coast had problems connecting to MailChimp on Thursday afternoon. Not Sandy related. DNS provider suffered DOS attack.
• Sun 11/4: Grid power (as opposed to backup diesel fuel) was restored on Friday in our NYC data center. It appears to be stable. We’ll likely be bringing those servers back into the mix starting Monday.