• We have 1.2 million users in 158 countries. That’s quite a growth curve since going freemium in 2009 w/100k users.
• Those MailChimp users upload an average 472,000 images per day.
• We serve about 115 million of those images per day (using about 3.5TB of daily bandwidth)
• Currently, we run MailChimp on 117 servers. 134 total, counting all our different services and products.
• We send between 80-100 million emails per day (using 3.29TB of bandwidth per day)
• Our servers track an average 20,305,881 email opens per day.
• We track over 4 million clicks per day

*| YOUTUBE:[$vid=XXXX] |*

that you insert into your MailChimp campaigns wherever you want to “embed” a video. If you’ve sent email newsletters long enough, you probably learned the hard way that embedding videos will break your HTML emails. To get around this, you have to take a screenshot of the video, open Photoshop, tweak it, insert it back into your campaign, and then hard-code the link. Which is a waste of time. Time you could spend photoshopping cats, or something.

Anyway, since introducing them in 2009, there have always been two complaints about our video merge tags:

1. People wanted more control over the look and feel of them, and

2. People who publish RSS-to-email campaigns wanted to make the tags automagically detect videos in their feeds, then convert them before sending the email.

Done.

Customizing Video Merge Tags

In the past, whenever we generated the thumbnail of your video, we added a border. In many cases, we also added the video’s star ratings, number of likes, number of views, and other “flair.” Now, you can take all that away and leave it barebones. Here’s an example:

On the left is how this Youtube video is embedded into your MailChimp campaigns by default. On the right is how it looks when we customize the video merge tag like this:

* |YOUTUBE:[$vid=XXX, $max_width=250, $title=N, $border=N, $trim_border=N, $ratings=N, $views=N]| *

You can tweak Wistia, Vzaar, Blip and Vimeo movies as well. Here’s the full documentation for that.

Automatically Converting Videos

These merge tags are cool, but where they really come in handy is in your RSS-to-email campaigns. For example, you can subscribe to receive daily emails from the MailChimp blog (ahem, signup form over in the right column). Our system checks my RSS feed every morning at 10am (that’s when I decided to schedule them) and if it finds new content, it pulls it into a MailChimp campaign. From time to time, I embed videos in my blog posts. I knew those videos would break when they got emailed, but I wasn’t about to modify the way I blog so that the email would work better. On the blog, the blog experience is what matters most.

Anyway, now I can just set my RSS-to-email campaign so that any embedded video it detects will be automatically converted into a video merge tag.

1. Open RSS-to-email campaign

First, I open the smart folder that shows all my RSS-to-email campaigns:

2. Pause campaign to edit

Next, click on “MailChimp Blog Updates” and then I get a list of all the recent blog updates that have been sent. The top one is the “source file” I need to modify:

3. Activate video conversion

Go to step 3 of the campaign builder:

and you’ll see all the different campaign setup and tracking options. In the lower right of the screen, look for “Content Controls” and check the box to autoconvert videos:

Hey, if you’re tinkering around with your RSS campaigns, this might be a good time to tweak your delivery time. Ages ago, when we launched RSS-to-email, we didn’t have the ability to pick a time or frequency. We’ve since added more control, so you might want to go to Step 1 and take a look at the options:

4. Save and re-activate the RSS campaign

Now that the video control option is set, click “Save & Exit” and you’ll be prompted to re-activate the RSS campaign:

Moving forward, any embedded videos we detect in your campaigns will be automatically converted to an email-safe format.

By the way, this automatic video conversion option works on regular MailChimp campaigns, too.

Related, IMHO:

• Build Your Own Content Digest With MailChimp – a tutorial we found by Jesse Rand
• We’ve launched tons of new email templates in recent months, many of which would be perfect for RSS-to-email campaigns. Check them out in the template gallery.
• If you’re a blogger, you probably have some VIPs on your list whose engagement you monitor closely. Our Golden Monkeys app can help with that. Get a push notification to your smartphone if/when a special someone opens your campaigns. Every time my blog post gets sent, my VIPs open and monkeys start screaming from my pocket. No better validation for writers than screaming monkeys.
• Other MailChimp features we think bloggers might like
• I’m often asked (usually by bloggers) for tools that can connect them to similar bloggers out there, so that they can promote each other and help each other grow their lists. Here ya go: Introducing Wavelength

That tool would analyze your list, then scour the vast database of MailChimp customers, looking for similar publishers to recommend. But this idea has been on the back burner for years, because such a tool would require 1) a vast database of MailChimp customers, and 2) the ability to analyze it–fast. Well, going freemium back in 2009 kinda helped with requirement #1. We’re at 1.2 million users, and manage over 800 million email subscribers for them all. And launching our Email Genome Project helped with requirement #2.

Helloooooo, serendipity. Finally, we have all the pieces we need to build Wavelength: a MailChimp service that uses a massive amount of email data to help you find publishers who share something in common with you:

Wavelength doesn’t help you send a promotion to another list, and it definitely doesn’t give you other lists or email addresses. It simply shows you screenshots of other newsletters that some of your subscribers read. The goal is to help you contact those publishers and maybe form a relationship with each other. Ideally, you can link to each other and help each other grow your lists organically.

How Wavelength Works

Basically, Wavelength analyzes your MailChimp list, then compares it to all other MailChimp lists (really, really fast). It looks for subscriber overlap, then recommends similar publishers by showing you screenshots of the email campaigns they’ve sent.

For example, let’s say I own a local pub, and I’d like to find some email newsletter publishers in town to partner with.

I’d go to Wavelength:

and authorize it to connect with my MailChimp account:

Wavelength will ask me which list to analyze, and it’ll ask for some descriptive tags for that list:

Then, it starts thinking:

It usually takes under 20 seconds to compare a list with about 1 million other lists containing 800 million emails.

And in order to deliver the results really fast, we pre-generated over 3 million campaign screenshots in the system (#NBD, as the kids tweet).

Once the analysis is complete, I get screenshots of email newsletters that my customers are also interested in.

They’re listed in order of “similarity” (subscriber overlap):

As one tester put it, this is where you meet all your “email cousins.”

From here, I can drill down to see an archive of past campaigns by each publisher, and then subscribe to any of their lists.

See your subscribers, and maybe even yourself, in a new light

The example scenario above is very typical for what we’ve been finding in our initial tests. You’d think that other local pubs would be listed first, but you’re more likely to find local theaters, beer-related iPhone apps, local coffee shops, etc.

When I ran my various MailChimp lists through Wavelength, I expected to see mostly email marketing or design related results. Instead, I saw that my customers subscribe to newsletters about social marketing tools, CRMs, content management systems, productivity apps, design publications, and newsletters about company culture and innovation.

Here’s a snippet of my newsletter’s wavelength:

But what’s really fun is when I manage different lists in Wavelength, I get some different results. For example, we manage a list that talks about our various giveaways (t-shirts, monkey hats, plushies, etc) that I think is mostly composed of very loyal (and obviously very stylish) MailChimp fans, and the Wavelength for that list looks like this:

Yet another list I set up for an event we hosted in London had a Wavelength like this:

which actually gives me some ideas for other international events to sponsor.

When’s this available?

We plan to open up access to Wavelength in about a month. Why the wait? Well, it scans our system for what it perceives to be public email campaigns, and it makes an attempt to exclude any email campaigns that it thinks are “private” (I’ll explain what that means below). But instead of just relying on algorithms to tell us what to exclude, we thought it’d be good to let our customers manually exclude themselves. We want to give you plenty of time to do that.

Public vs. private email campaigns

MailChimp was built for email marketing, which is an inherently public activity. So what in the world should be considered a “private” campaign, and why would someone use MailChimp to send one?

Usually, it’s an internal company newsletter, or a wedding invitation, or a one-time prize notification or transactional kind of message.  The information in the email is not usually super private or sensitive (email is just not an extremely private medium), but it might be something that you don’t exactly want promoted, or something with expired content. Wavelength will almost always exclude these, because it won’t search lists that were only imported manually, it won’t include tiny lists or fresh new lists, and it won’t show campaigns sent to a segment of a list. For a campaign to be shown in Wavelength, its recipient list must be greater than 200 members, and show signs of being public. Namely, opt-ins were received from its public signup form, or the campaign archive bar (that thingy with all the social sharing buttons) is activated.

But if you want, you can manually override everything, and totally exclude your list from Wavelength searches.

For example, I have my list where customers can sign up for a chance to win a t-shirt. For some reason, I just don’t want this to show up in Wavelength results. Maybe the t-shirt designs are top secret prototypes or something. For that list, I can go to “Publicity Settings:”

And then mark its campaigns as private:

You’ll notice that while we were at it, we combined two other previously released features that have publicity and privacy options (the archive toolbar and the subscriber count chiclets). We figured it’d be nice to consolidate everything in one place.

Using data to make email better

Wavelength is a project I’m happy to finally see the light of day, but we’ve only just begun. In 2011 we brought on a server/devops guy to help us handle all this “big data” and we hired an internal data scientist to analyze that data (here’s some fun stuff he’s found). We’re already heavily using EGP behind the scenes here to prevent abuse and protect the email ecosystem. For example, about a year ago, a hacker stole someone’s identity to create a MailChimp account, then used it to send spam (one reason we’ve added so many security features to MailChimp, and why we make free 2-factor security apps like AlterEgo). After that incident, we analyzed their list and found other accounts that had lists very closely matching the hacker’s:

The “evil doer” is in the center, with similar lists surrounding (users’ names obviously have been obfuscated). See any common theme here? What we found was fascinating. Some of the “similar” accounts were legit users, and some appeared to be accounts that the hackers were in the midst of setting up. But this kind of graph raises questions like, “Why are they all London arts / entertainment organizations? Did they initially steal their list from some London theater? Or did they all scrape their lists from the same source?”  We’re also able to test incoming new accounts for the presence of stolen/purchased/scraped lists (based on data we’ve accumulated from accounts we’ve shut down for abuse), with the goal of keeping our system clean and our deliverability high (and also, you know–protecting email). And most exciting of all (to an email nerd like me), we can use what we’ve learned while fighting abuse to build cool new features like Wavelength that help us improve our email marketing. Who knew math could be so useful?

To be notified when Wavelength goes live, sign up to this list.

For general announcements about our Email Genome Project, subscribe here.

Related reading:

• Fun with data science (and dendrograms): http://blog.mailchimp.com/fun-with-data-science/
• Slightly scary: What happens to email typos? http://blog.mailchimp.com/comacast-and-gmai-all-your-typo-email-are-belong-to-us/
• Pesky tweets from scantily clad fembots: http://blog.mailchimp.com/dealing-with-tweets-from-scantily-clad-fembots/
• MailChimp’s Email Genome Project: http://blog.mailchimp.com/mailchimps-email-genome-project/

Planned Downtime: January 22, 1am ET

First, we’re doing some server maintenance at our US1 data center on Sunday, January 22nd at 1am ET (see this in your timezone). The maintenance will require downtime, but should only last a few minutes. During those few minutes, MailChimp will not be available for US1 users at all. Their campaign links will not work, nor will new subscribes be tracked. Again, it should only be a few minutes before everything’s back online. This upgrade will basically help us rebound faster should a similar outage occur again (heaven forbid).

So what exactly happened that day?

To recap, last year we invested in super fast SSD equipped servers to handle our increasing traffic. They helped us handle a TON of load, and sped things up nicely through the holidays. Then on January 2nd, several of those servers just up and died all at once–for no apparent reason at all. It just didn’t make any sense, and we’ve never experienced anything like this before. We admittedly didn’t spend much time investigating the cause, because we were busy taking out those SSDs and replacing them with 15k rpm SAS drives (plus a bunch more RAM).

Then a few days later, we saw this news: 64GB Crucial M4s crashing after 5,000 hours, fix coming

Those were the exact drives our data center used, and 5,000 hours is how old they were. We can’t say with 100% certainty that was the cause, but we can say there were other drives at US1 that didn’t fail, and they were different models (if we were the cause, you’d think all the drives would fail). And as stated above, we’re making some changes that should make events like this faster to recover from.

Some people have asked us, “So does this mean you don’t keep frequent backups?”  And from our close friends who know better, “Ha, guess you guys don’t know much about redundancy?”

To shed some light on the insanity of the situation, 76 hard drives died in 6 hrs. Fortunately, since we separate our users across 3 different data centers, the majority of them had no idea anything was even wrong.

For anybody who’s curious, there’s also this comment from Joe, one of our DevOps Engineers:

I thought I would briefly mention how we set things up though as we take these sorts of downtime and data loss events very seriously and I don’t want people to think this was a simple “a machine died, we restored from backups” scenario.

Each user shard in US1 is supported by 3 separate machines. Each of these machines is powerful enough to easily support the entire user shard on its own but only one of these is active at any given time. The other two sit mostly idle staying in sync and ready to take over in the event of a failure. We essentially are always running up to the second backups to two different machines for each user shard. In addition to all of this, we run full backups on every shard every day for “disaster” scenarios.

We very, very rarely lose an entire machine and when we do users do not notice because one of the backup machines is activated automatically and takes over.

In this case the nature of the hardware failure was so severe and so catastrophic that it impacted ALL of our machines – causing them to crash and corrupting their entire raid arrays of SSDs as they went down. We were unable to pull the current data off these machines for all shards before they were fully offline and despite spending many, many hours trying to recover the data after the crashes we failed except for that one clump of users that didn’t have any data loss.

Hope that sheds some light on our process. This was a truly unique event that I hope never happens again. We are making changes now to fortify our shards against data loss even further.

Customer feedback from the outage

I wanted to share something else with you. After the outage, we sent an email apology out to 788 users who were affected the most severely by the outage.

Here’s an archive of the email we sent.

I asked those customers to reply and send me any feedback they wanted. I totally expected to be screamed at and threatened for the next few weeks.

Instead, all I got was positive energy. People told me they loved us anyway, that my health was all that mattered, I’m a good human, etc. Here’s a sample of the replies I’ve received so far:

—-

So I am writing to thank you for you attention to this situation! I still LOVE Mail Chimp.. no worries… It was frustrating to reconstruct my campaign and to resend to my list… but I did it and all is well. Luckily, I had sent the campaign and could reconstruct it from the one I sent to myself.

—-

Ben, I’m so bummed. ARGH. Oh, that totally sucks. I thought that I had only lost a mini add-on campaign to one I had done earlier, but I just discovered this evening that it was the whole kit and caboodle campaign. SHEESH! The good news is that I’ve gotten some email replies from the campaign it looks like it got sent, but I have no way to know that for sure, who opened it, who doesn’t want to get email from me anymore, etc. as it doesn’t show up as a sent campaign on my dashboard.

Flippin’ firecrackers, I’m disappointed! I kinda want to throw a banana at you. BUT you’ve got such an amazingly fantastic, user-obsessed, FREE product, it’s hard to be mad at you.
How can you win back my confidence…? I would really love to know the stats for the campaign I sent out. I would like people from that to unsubscribe without sending me an email. Since those things are probably impossible, there’s not much else I can ask of you. You made a mistake, you totally owned it and are doing everything in your power to make it better. I was a Mailchimp evangelist before this, but I daresay this little f-up and your response to it might have just elevated my opinion of Mailchimp even higher.

Thanks for doing and being your best.

—-

Ben, thanks for your lovely email! I really feel you guys care about what you’re doing
Anyway, it’s no biggie. Thanks for letting me know.

—-

Hi Ben,
I don’t need any kind of compensation. This apology is more than enough for me.
Thank you!

—-

Dear Mr Chestnut,

Greetings from the Czech Republic and I hope you are doing well!

I am regret to hear about your hardware failures that have an impact on us. We have a good experiences with Mailchimp and it is clear to me that these things could happen. Due to the effort, the discount for the next month would be much appreciated! Kindly please let me know if it would be possible.

I am truly looking forward to hearing from you and wish you and your family all the best for a healthy and Prosperous New Year.

—-

Hello, thank you so much for all the explanations. As you say, we couldn’t send our campaign yesterday, but there is no problem with that. We’ve been using your service without any problem until now, so thank you for that and don’t worry about your hardware failure. These things sometimes happen.

—-

Although I lost my campaign and as a result need to send it again because of this failure,
Your truly honest mail and apology is something rare in the business world, and for that you have my full confidence in your service,
and much more important, in your credibility as a human being that doesn’t afraid to admit he made a mistake.

So, Thank you for this wonderful service of yours,
and apology accepted!

—-

I just wanted to say that you guys are awesome. Mailchimp customer service is SO top notch every single time. What a perfect “I’m sorry” email  

Love y’all, and no worries on the campaign. I’ll just shoot out another one. 

—-

I just got you email – as a free user I am ok with having a simple outage in fact all I lost was the report for a campaign I sent out on the morning of “the incident”. And as much as I enjoy seeing which of the 46 magicians read our email reminder – I’m sure the world isn’t going to end.  I say all that to say I appreciate your integrity and willingness to go the extra mile – it makes me confident that if my website ever gets off the ground – I’ll use mailchimp as my email service – thanks for being AWESOME.

—-

That is ok, Nobody can controle the lectronic world….if your healthy..that is all that count

—-

Hey Ben,

No biggie, it was just one of those automated RSS campaigns. This isn’t my paid account which I use for another project so I appreciate the awesome free level of service you continue to provide. You guys do a great job keeping me informed and providing a reliable, easy to use service with helpful support staff. Keep it up.

—-

Hi Ben,

Thank you for your message, I appreciate your concerns. At this point I am only doing testing so nothing of any significance was lost. This is my first attempt creating an RSS campaign and I must say I am quite impressed. I think MailChimp is great.

—-

Thanks for owning up to the problem. Ironic that it happened on the first business day of the year.  

Mailchimp is a great service. Thanks.

—–

Glad to here all sorted guys. Would prefer a t shirt via they are way cool.

—–

I’m devoted to MailChimp in the same way I’m devoted to Apple, Gmail, WordPress and Saddleback Leather. You’re a brand head-and-shoulders above the rest. When I was unable to access my account yesterday, it was disappointing, but had you not sent this email I’d have thought nothing of it. MailChimp had never disappointed me before, so one day without access or autoresponders hardly bothered me.

I appreciate the gesture, though. Keep up the excellent work, Ben. My businesses couldn’t run without you guys. Hopefully soon I’ll have enough subscribers to use the $50 credit on the premium service. (Although, a free lifetime account would have been an awesome way to “make things right” )

—-

Dear Ben,

Thank you for informing me so openly – i really appreciate it.

You are doing a great job and i am very grateful for your services.

My campaign link was broken, but no problem!

Many thanks for your support.

—-

Crap happens. You’all are still my favorite chimps. Thanks for the heads up.

—-

It happens. Don’t worry. I have full confidence in mailchimp. My campaign went and that’s all that matters. I don’t need a record.

Thank you for the explanation.

—-

Thanks for the email but really it seems that my email went out just fine. I got a copy in my inbox and everything looks Okay to me. We are still loving MailChimp!

—-

There were a couple not-so-happy ones too (and well-deserved) but the overwhelming majority of replies were very positive. I love our customers.

Here’s what’s rolling out:

• Automatic video conversion for RSS-to-Email campaigns
• Discount on MailChimp if you use it with AlterEgo
• Static segments: create segments fast by uploading a list of email addresses
• Daily Deals is now an industry category
• GMT +12 timezone support added. Welcome to MailChimp, Republic of Kiribati!

More details and documentation will be posted soon, but if you’d like a general overview…

Automatic Video Conversion for RSS Campaigns

Most email applications just won’t reliably display embedded videos within HTML email (because email apps pretty much block any code that might pose a virus risk). That’s why we created our video merge tags back in 2009 . These are great whenever you’re manually building a newsletter, but what if you have an RSS-to-email campaign that automatically grabs content from your blog? Nobody wants to stick special tags in their code when they’re writing a blog post (I sure as heck won’t). So we made MailChimp automagically detect videos in your RSS feeds, then insert our video merge tags for you. There’ll be a new checkbox in the campaign creation process that activates this feature. More details, plus some cool new video merge tag options, will be posted soon.

Update: apparently, our engineers made this work for ALL your email campaigns, not just RSS ones. Win!

Discount For Activating 2-Factor Security

We’re rewarding all our AlterEgo 2-factor security users for being good, security-conscious netizens of the email eco-system by giving them a 2% discount on MailChimp. (Get it? 2-factor security -> 2% discount?) If you manage a large list in MailChimp, this discount can be a pretty significant savings for you. And 2-factor security just might be able to save you from email breaches (which is significant no matter what the size of your list). If you’re an agency who manages clients in MailChimp, you should consider setting up AlterEgo for your clients, because it’ll make you look pretty darn responsible, and save them a little money.

Upload Your Own Segments

MailChimp’s built-in segmentation builder is pretty easy to use, but some of our more advanced users just want a super-fast way to manually upload a list of email addresses to build a segment (like a “do not email” segment, or maybe some complex segment that only they can build with their proprietary e-commerce systems). Since these aren’t dynamically constructed in our app, we call them “static segments.” Most people build these and then upload to MailChimp via our API, or with our new Hairball app. Well, in v6.7 we’ve exposed this functionality within the MailChimp interface for anybody to use. Screenshots and tutorials are in the works.

Daily Deals Industry

We thought it might be cool interesting for geeks if we did a wrap up of 2011 at MailChimp. One thing that came to mind was how many Groupon-style “daily deal” sites signed on to our service. There was definitely a trend there. We want to do an analysis of how this industry performed (anonymized and in aggregate, of course). However, we first had to make it an actual industry in our system (for those of you who don’t know, specifying your industry will show you how you compare to your peers):

If you’re in the “daily deals” industry, make sure you change your category (it’s under your contact information in MailChimp) so you can be included in our analysis.

GMT +12

We received a request from a user in the Republic of Kiribati to add their timezone in our options. Since we’re on a mission to spread our monkey love to every corner of the globe, we thought we’d add this really quick. Also: http://en.wikipedia.org/wiki/Kiribati

“Twitter malware and spam uses a pretty straightforward attack vector. You get a twitter message from an account (usually with an attractive female avatar) telling you that you’ll get something awesome if you click on the helpfully provided link. Most people don’t click, because they realize that if a hot chick sends you a link on twitter claiming you’ll win a free iPad, it’s probably not legit.”

The author goes on to predict that twitter spam will get a lot more sophisticated and targeted, and it will get harder and harder for people to determine who to trust and who’s a bot (speaking of bots) and who’s not a bot:

“Twitter link spam will get a lot more context aware in 2012 and it’s going to be difficult to make an eyeball determination whether someone you don’t know has sent you a link because they follow you and they think you will be interested in a topic, or they are just trying to spam you”

As a matter of fact, we get a lot of tweets from scantily clad fembots that try to make us click malicious links, so we built an app to deal with that. It’s called Unfurlr, and you’re free to use it too, whenever the fembots come knocking –>  http://unfurlr.com  (bookmark it now, because they will come knocking)

And here’s a little more background info about Unfurlr.

On January 2nd (yesterday), we started to see multiple hardware failures at one of our data centers. As background, we’ve spread MailChimp across 3 data centers across the country so we don’t have all our “eggs in one basket”. Then, we further divide each data center into different groups or “shards” of users. Some shards house big, high-volume users with large lists and intense server resource requirements, while some shards are for users with relatively smaller lists (less than 25k recipients is considered “small”). This is an attempt to keep issues for one set of users from bringing down our entire base of 1.2 million users.

US1, which is our first and oldest data center with the most users, saw 3 of the “small user” database shards failing. Around 2pm, we decided to completely disable access to those 3 database shards in order to prevent those users from logging in and creating new campaigns, which would’ve been lost in the event we had to restore from backup. This affected about 400,000 users.

We then began the long, painstaking process of replacing hardware and then restoring data. For 1/3 of the affected 400k users, we were able to restore things nearly perfectly with no campaign or data loss (note that during the outage, we could not collect any new subscribe or unsubscribe activity though). For the other 2/3, the failing hardware corrupted data so badly that we had to revert to backups from 1am ET that morning (January 2nd).

So for those users, any campaigns created, or campaigns sent (including all tracking links) between 1am ET and roughly 3pm ET were lost (RSS and Autoresponder campaigns, however, will simply pick up where they left off).

We’ve researched the logs, and it looks like 788 users lost that campaign data between 1am and 3pm. Needless to say, an email with apologies and refunds and make goods for these users is already in the works. If you were affected by this outage, do not hesitate to contact our support team. They have full authority to make things right with you.

We’re still investigating the exact cause, but we know that back in September, we began replacing our older hardware at US1 with brand new, super fast, super expensive SSD-equipped servers in order to beef up for the Thanksgiving and Christmas holiday email volume. This was a significant infrastructure investment for us to keep things stable (yes, the irony). The upgrades did manage to sustain delivering +100 million emails per day during peak periods, but the RAID controllers for the SSDs weren’t working as reliably as we hoped. When those things fail, they apparently break the SSDs along with them. So our plan was to switch back from the SSDs after the holidays when things are usually more stable and quiet (yes, irony). Obviously, we’ll be accelerating those plans as soon as the dust settles.

We sincerely apologize for this, and will be working extra hard to regain your confidence.

P.S.

If you’re into gory technical details, read the comment from Joe, one of our devops engineers, below.

——–  below is the original post, which I wrote in haste to keep people updated —–

Hi everybody, sorry to report we’re experiencing some hardware issues at our US1 data center, which is affecting a large number (but not all) of the users there. US2 and US4 users are not affected. We’re currently in the process of doing some quick final backups, so that we can replace the hardware. To prevent data loss during the switch, we’ve disabled access to MailChimp for those users. We expect this process to take several hours. We’re very sorry for this inconvenience. If your campaigns are affected, talk to our support team, and they’ll work to make things right with you. We’ll also be posting updates here on the blog and on twitter at @mailchimpstatus

Update January 2 (10:28 pm): We’ve replaced the hardware, and are currently reinstalling software and data. This is going to take several more hours to get fully operational.  We’ll be working through the night to get it all running smoothly again. Sorry again to all affected users.

Update January 3 (4:50 am): Slight progress. 25% of affected user accounts have been restored and are back online. Still working on restoring backups of the remaining accounts.

Update January 3 (7:47 am): A little more progress. Another 25% of the accounts have been restored and are back online.

Update January 3 (8:24 am): The final batch of users have been brought back online. From our customers’ perspective, US1 is live again. Behind the scenes, we’re mopping up some messes, and soon I’ll post more details here on the blog.

Sometimes they’re a little more excited than normal:

At first, I thought the summer heat in June and July took away all their excitement. But you can see on the next line they snuck a little excitement back in:

They were just “pleased” in June:

Honestly, for a frequent system alert email like this, they know what they’re doing. A simple little logo, plus some text is all they really need, and is all we’d put up with. If the started getting too fancy-schmancy with their designs, I think people would get tired of them really fast. These seem to just fly under the radar. In fact, after watching them send these all year, it motivated me to stay in touch with our own customers (a little) more often with our System Alerts. But since they do send so frequently, it makes it a lot easier to spot little details like this. I started to notice it about halfway through the year, and decided to just keep checking to see if they’d ever change it up. At this point, I hope they don’t. It’s grown on me, and I think it just wouldn’t be the same if they weren’t excited anymore.

Update: Be sure to check out the awesome comment from Amazon’s Jeff Bar below. I was truly honored (and excited) to see he took the time to reply to my silliness.

We’re in the midst of upgrading our servers. The goal is to help make things nice and fast and stable for all the holiday traffic we’re expecting for our senders.

Unfortunately, it looks like there may be certain scenarios where relatively complex segmentation queries caused us to send out duplicate emails. This is obviously something that’s potentially extremely embarrassing for us, but more embarrassing for our customers, so we wanted to get this apology posted for you to link to in case this affected your campaign. As of right now, we’ve only seen one report of this happening (but in a big way, unfortunately), so we’re scanning the system to see if there are more cases of this. If you don’t send campaigns that use relatively complex segmentation queries for different interest groups, there’s nothing to worry about. If this affected your list, please let our support know. They have authority to make this right for you. If you’re a subscriber who received duplicate emails, we’re extremely sorry about that.

Read the bullet points. Particularly, #2 and #5:

When you’re writing your emails (especially automated inventory alerts like this one) you can get away with boring, functional, corporate writing and nobody will mind. But can you get away with sneaking in a little fun? That’s a lot more challenging imho.