Every day, about 5,000 people sign up to use MailChimp. When they do that, we ask if they’d like to opt-in to a Getting Started email series to help them become power users, and about a third choose to subscribe. If you signed up for this “autoresponder” a few months ago, you probably saw a similar prompt labeled “Boot Camp.” But recently, we’ve been experimenting.

Back in the day, “MailChimp Boot Camp” was a set of 10 emails we automatically sent a few times a week to the people who checked this box, reviewing the power features that MailChimp offered. The training was going well, but after a year, we thought it’d be worthwhile to see how we could make it even better. Plus, we know that in order to help our users improve their campaigns, it’s a good idea to eat our own dog food first.

Some users have evidently noticed our recent changes, and asked if we could talk about our approach.

The Fix-Up: Research and Revision

To rewind a little, I should introduce myself. Hi. I’m Brad. I’m a researcher on the MailChimp Support team, and I’m kind of a math nerd that likes analyzing numbers. People around here have apparently noticed my fondness for stats and experimentation, and asked me to take a stab at improving our Boot Camp training series. I had no idea what to expect, so the first time I logged in to this account, my eyes wandered to all corners of the screen, expecting complex API-based triggers, an embossed MailChimp monogram, or at least a cool background midi from an ’80s movie soundtrack. Fortunately, Boot Camp was sent using everyday MailChimp autoresponders. Turns out, a standard MailChimp account is more than up to task.

First, I checked the reports to see what people loved and what they didn’t. Right off the bat, I noticed features that were buried at the bottom actually outperforming more prominent links, so we thought we should try moving them up to get them in front of more users.

Next, with open rates leveling toward the end of the series, we thought it would make sense to cut the 10-email series in half.

“Choose Your Own Adventure”

But instead of scrapping the extra stuff altogether, our marketing team created newsletters for seven different verticals. If you’re still aboard for that fifth Getting Started email, we ask if you’d like to stop, or continue on with a specialized set of 4-5 more based on your industry. These emails reintroduce some advanced topics and speak directly to bloggers, nonprofits, restauranteurs, etc.

Rebrand and Redesign for Mobile

Fabio whipped up a new template (optimized for mobile!), and we retired the “Boot Camp” title. We were trying to make things easier, after all—not scare people away. Qwikster and New Coke were already taken, so we kept it simple with “Getting Started.”

When new users mark the Getting Started subscription box, they’re added directly to a list in our MailChimp account. For the initial sequence, five autoresponders were set up to send to this same list. After the first autoresponder was completed, the new template made it easy to replicate a campaign and simply update its send settings to build a totally automated progression that delivers a new email every three days.

The Engagement: Facts and Figures

The refreshed version has now sent more than 800,000 total emails, and the changes have been a hit!

Right off the bat, the open rate for the first Getting Started email is approaching 50 percent with almost 40 percent opening throughout the sequence. And twice as many people are now clicking at a clip of nearly five percent. Looking at our industry averages for click-through rates, that’s not too bad.

The industry-based emails are doing even better. And I think it’s fair to say this is an excited bunch. Combined, there’s a 55 percent open rate and eight percent click rate. 74 percent of musicians are opening their first email, and more than 12 percent of online sellers are clicking.

Forking the series and allowing our users to specify their “industry vertical” was the biggest change we made, and we think it was the best. The verticals have much higher engagement than our general-audience ones. Instead of sending generalized information to the masses, we can send more relevant messages. It’s easier to write content this way, too. Another benefit to this approach is that if we ever launch a new feature for one of these segments (nonprofits, for example) we can easily insert it into that sub-series. Although, we may need to re-think the “Issue #1,2,3…” sequential approach in our template design. These are all things you might consider with your own autoresponders.

Here are the open and click rates for each of our autoresponder series. But even with a positive response, we’re reviewing our content regularly and looking for ways to boost these stats further.

The Ugly Truth: Mistakes Were Made

One thing you might notice after looking at the images above is the drop off from Getting Started to the vertical sets. In total, as the volume of Getting Started approaches one million, we’ve only sent around 14,000 vertical emails. Here’s a graph comparing the total sends of the main training to the advanced.

Only about one percent of people who are sent the fifth Getting Started choose to keep going. If we’re just talking about the people who open that email and are able to read about all the things our continuing emails have to offer, this number is up near three percent. There’s a useful marketing term called “funnel,” where this rate is okay, but when I first saw the loss of subscribers (look at the graph!), I tried to karate chop my desk in half, resulting in a swollen hand and a fully intact desk. Stupid, durable desk.

We all met and looked at the autoresponder where subscribers can choose to keep learning, and decided that this email has some usability problems. Without a clear call to action, you have to look deep inside a block of text to figure out where to sign up. The screenshot we used is basically an “ornamental” image acting like a “content” image, when what we truly need is a “navigational” image (here’s a good usability article on image types). We need a graphic that directly guides readers to our additional emails with no confusion or opportunity to hesitate.

We also discussed the original choice to stop the series, and force readers to opt-in for more. Why not just keep it going, and let subscribers decide when they want to get off? Maybe we can just modify the #5 email to simply tell readers that “things are about to get more advanced” and offer them an unsubscribe link if that’s scary to them.

This issue is one of many that we’ll be looking to reform in the coming weeks. We’re happy with the results so far, but know that there’s still a lot to be done if we want to keep elevating users.

The Big Salad: Moving Forward

Anyone can make a few small changes for a big impact. Here’s a list of tips based on our experience:

• Start with a goal. Ours was “turn users into power users,” which is different from “onboarding” or “make free users convert into paid users.” We learned that when more people contribute to the project, the goal of your automation project can get off track.
• Review. Once you’ve started sending, don’t forget that reports can be as important as the original campaign.
• Revise. If something doesn’t seem to be working, try something new. If you’ve completed the Getting Started series, browse some great campaigns for inspiration, or feel free to ask us for advice.
• Have no fear. Don’t be afraid to shake things up.

Even with the success of these updates, our work here isn’t done. To help guide future changes, we’re gathering feedback with ReplyTo and tracking subscribers with ListMD. And we’ll keep trying to improve with help from more of our own tools along the way.

Most importantly, Getting Started exists to empower. In addition to helping users build their list or upload a first photo, it serves as an outlet to demonstrate how useful our Google Analytics integration can be regardless of your industry or existing email experience. While MailChimp is filled with tons of advanced features, “advanced” doesn’t have to mean “difficult.” Throughout these autoresponders, we hope not only to display the abilities of our app, but also to show how easy it is to succeed.

If you want to see these autoresponders up close, sign up for the Getting Started series. Or go straight to one of the vertical series. And all of the emails are available in the MailChimp Email Archives. Now, I’m off to enjoy some relaxing waves beneath a Hawaiian sunset. Ahhhhh…