Jan 15, 2008

Case Study: MailChimp API Campaign

api_thumb.gifBack in early November, we released a totally revamped MailChimp API. It’s a huge upgrade. So we wanted to send out an email campaign telling our customers about it. But we didn’t just jump into this one. We planned this one out (that’s rare for me).

I thought I’d document the process and the results here on the blog. Who knows, you might find some of it useful…

Background

The new API was finished around mid-November. It’s so advanced and powerful, we wanted to get the word out immediately. However, the Thanksgiving holiday was just around the corner. Our website traffic was already showing signs of people "checking out" mentally. Our marketing team knew that logically, sending out an email campaign and press releases over Thanksgiving would result in a whole lot of nothing. But our technology guy mentioned the fact that "This is an API we’re talking about here. Nerds like me are the target audience. And nerds program over the holidays."

Hmm.

The Plan

So we figured instead of sending out one big "New Feature" blast to our entire user base, we should direct our marketing at two different audiences:

  1. The "Nerds" (and we mean that in a good way) who would ultimately program the API integrations, and
  2. The rest of the people who would ask the nerds to "please program this API thingy" for them.

We named the campaigns "Nerds Only" (Campaign #1) and "The Rest of Them" (Campaign #2).
For the nerds, we decided to send a campaign just before the Thanksgiving holiday (Wednesday, November 21st). Then, in between tryptophan-induced naps, they could spend a little time tinkering with all the XML and JSON goodness.

For the rest of them, we scheduled the campaign to go out Friday, December 03 (plenty of time for them to get back to work).

Our previous API was caveman simple. It needed lots of example code, and we wrote the documentation so that a "dummy" could understand it. This had a negative impact on customer service, because indeed, lots of dummies ended up calling in. Not just people who needed a little help with some code. That would be fine. But we had people who had no business even using computers calling us and asking us to program their APIs for them, plus a website, plus an "email blast."So customer service asked us to create a better landing page, that basically suggested that this API stuff was for advanced people only.

Marketing wanted a landing page that would inspire our non-technical people to consider the API, then hire an expert to get it done for them (but we didn’t really have an "experts" list to point them to—yet).

We decided to create a landing page that was split down the middle. It poked fun at the "Venus and Mars" relationship between marketers and techies, meanwhile explaining the potential power behind the API.

The Email Creative for Campaign #1

For the nerds, we figured a text-heavy, "just-the-facts" approach would be best. Techno-babble is actually a good thing in this case. We stuck with our clinical and functional MailChimp Email Alert template, and just loaded it with text and examples.

Here’s an archive of campaign #1 (for the nerds)

We also created a MailChimp "Experts" directory, where these techies could get their names and websites listed, so that our non-technical customers could have some freelancers to call on (we were going to need this for campaign #2). To encourage signups for the directory, we offered two prizes: a Mozilla Firefox laptop bag for the first entry (Mozilla had recently started using MailChimp, which is a nerd dream-come-true in itself), and free t-shirts for the next 10. The t-shirts were from ChopShop, who also uses MailChimp. Their shirts appeal to the freelancer/creative/nerd crowd. In order to take all the entries for the directory, we used SurveyMonkey (we didn’t have time to program a database that plugged directly into our website, and SurveyMonkey does just fine for us).

The Email Creative for Campaign #2

For non-techies, we needed something a little jazzier, eye-catching, and thought-provoking. These guys would never understand the "XML, PHP, JSON" jargon. But they’d totally love to read about the behavioral targeting and metrics possibilities with the API. So we kept the content in plain English, and tried to emphasize benefits more than features.

We created a custom campaign design with a huge animated graphic as the centerpiece. Then we took off the stupid hat, and scaled it back to fit it into one of our more realistic, built-in MailChimp templates (because we knew those were field-tested to work).

Here’s what campaign #2 looked like (for the rest of the list)

Tactics & Results: Segmenting and A/B Splits

Campaign #1 (Nerds only)
For the techies, we segmented our customer list with the following criteria:

  • They’ve logged in within the last 6 months (and so would be considered "active")
  • They’ve classified themselves in our database as "Programmers, Freelancers, or Web Designers."

Then we created our campaign and ran an A/B Split test by subject line. 10% of the list got subject line A. 10% got subject line B. Whichever subject line got the better open rate within the first 6 hours, that’s the "winning" campaign that would be delivered automatically to the rest of our list. We picked 6 hours because we were pressed for time, and we’ve found that the majority of your list will open within that timeframe anyway (Study: Open Velocity).

We experimented with subject lines that were short and sweet, versus long and full of jargon:

  • Subject (A): Sync your external dB with MailChimp’s new API
  • Subject (B): API Upgraded: XML-RPC Documentation, PHP samples (nerds only)

The winner within that timeframe was Subject A, but only by a tiny margin: 44.44% vs. 42.22%

The campaign was delivered to the remaining 80% of our list using Subject(A), and it received a 45.35% open rate.

At the same time we sent the campaign, we posted this information to our blog (techies are totally into the RSS thing).

Campaign #2 (the rest of them)

The second campaign had more "pretty pictures" and was designed for our non-techie audience (basically, the rest of our customer base).

We ran an A/B experiment with subject lines on this group as well, and the results for this group were way more dramatic than the ones for techies:

  • Subject (A): Sync your company database w/MailChimp (33.33% open rate)
  • Subject (B): MailChimp API upgraded (40.46% open rate)

The timeframe for this campaign’s A/B experiment was four days (as opposed to 6 hours for the "nerd" campaign). It’s important to note that people check email months after you send your campaign. So going back to look at stats now, we see the difference in open rates is 36.31% vs 45.22%. Anyways…

The Outcome

One regret we have after all this is that we didn’t program anything to tell us "how many people are actually using the API." Hey, we work fast around here. Sometimes, too fast. All we have are raw traffic logs for "how many times the API has been called." Those logs are over 700 megs, which amounts to slightly over 4,000,000 requests between mid-November and January 15th.  Analyzing them is low on our list of priorities right now, but we hope to come back and update this case study with more detailed numbers.

But here’s what we got out of this whole campaign:

  • The "Nerds Only" campaign resulted in extremely valuable, positive feedback, within hours of sending. Talk about "early adopters." Feedback was along the lines of, "Oh God yes, thank you" and "I’ve been waiting for this for years"
  • The techies also found a couple bugs that we were able to take care of before the rest of our customers found them. Their input helped us put together even more documentation before launching to the general public.
  • Customer service received several, "I’m switching to MailChimp, because of this new API" messages.
  • One of the recipients on our "techie" list was in the middle of writing a web development book, and he’s now mentioning our API in that book (to be published in February, but that’s a different blog post for later).
  • Traffic to the API landing page tripled on the day we sent the campaign, and quadrupled on the Monday after Thanksgiving. Traffic stayed strong throughout the rest of the week.
  • This one’s difficult to quantify, but interest from very large corporations is at an all-time high.

Observations and Advice:

  • This took a lot of work and planning. If you do this for a living, you do not get paid enough. Seriously, you deserve a raise. We normally send a quick campaign out to all customers making product announcements. Or maybe a monthly newsletter. Our MonkeyWrench newsletter takes a lot of research and writing, but never anything this elaborate. It takes a ton of work. Plan ahead.
  • Since we used built-in templates, it kept the emails fast and simple. I’d say 98% of the work goes into "content your email links back to" such as the landing page, experts list, and API documentation.
  • Don’t just automatically assume that sending on a holiday will get you nothing. There are lots of people out there who check email (and work) on the holidays. Techies, entrepreneurs, creatives. They’re "always on." The key is segmenting your list to target them.
  • For the techies, we thought the jargon-heavy subject line would win, but we were wrong.
  • For the 2nd group of "the rest of them", we knew the simple "MailChimp API Upgrade" subject line would win, but we’ve always been told to write with benefits in mind. Gladly, we were able to run a simple A/B experiment, instead of wasting all our time on the ineffective subject line.
  • The first campaign was sent on Nov.21st, 2007. It received an "open" most recently on Monday, January 14, 2008 at 3:49 pm. People will check your email months later. Keep an archive.
  • Always segment, and always experiment.