Apr 14, 2014

Experimenting with MailChimp’s Quarterly Newsletter

We’ve been sending our MonkeyWrench newsletter every 3 months since 2006. In the beginning, we had 136 subscribers, Gnarls Barkley was topping the charts, Pluto was enjoying its last days as a planet, and a group SMS app named Twttr was still missing its vowels.

Not everything has changed, of course. We continue to deliver tips and tricks in issues built around a theme (the next one is "experimentation"), but now we’re able to present our own research and share the experiences of our customers. We’ve also tweaked the design quite a bit since the early version in that screenshot above.

Still, the MonkeyWrench list is old. Fifteen of our now nearly 150,000 subscribers use @prodigy.net email addresses—yipes! Many other discarded addresses have expired, bounced or have been automatically cleaned, but others lay dormant. Some members have lost interest, but just never unsubscribed.

Recently, we started thinking about these people and doing some experimentation of our own. If they remained inactive, we’d know it was time to clean our list. If we could find a way to get them to open again, we could use these strategies to improve the response from everyone else.

Removing old subscribers

First, we isolated everyone who received the last 4 campaigns (a year’s worth of MonkeyWrenches) but didn’t open any of them. This ended up being about a third of the list. While the effect of a bulk unsubscribe would likely be minimal, we kept these idle subscribers in a static segment, just in case.

With that group excluded from our next send, opens improved from 25% to 36%. I did some quick math, rewatched Jurassic Park, and figured the expected open rate from those inactive addresses would’ve been about 5%.

Ladies and gentlemen, Kevin Hillstrom

A few weeks later, Kevin Hillstrom spoke at MailChimp’s Coffee Hour. He’s written a book about email subscriber behavior, and managed email marketing programs for large retailers like Lands’ End and Nordstrom.

As part of his talk, Kevin suggested using inactive members as an opportunity to experiment. He explained that, since these folks aren’t engaging anyway, the possible negative side effects of experimenting with them are low. Tests are hidden to regular readers, and if a strategy works, it’s easy to apply to the rest of the list. This seemed like a good idea, so we gave it a shot.

We decided to try experimenting with our content, shortening and simplifying each section. In February, we sent regular readers our standard campaign, but split the inactives in half. The first half received a replicate as a control. The other half got a shortened version, where each article was listed with a title, link, and one-line description. We didn’t expect many people to open either one, but figured we’d gather some good click data from the ones who did. The streamlined style would drive more clicks with less text, and a 50% click rate was in sight.

What to expect when you’re not expecting

Of course, more people clicked the regular campaign. Looking at clicks per unique open, the standard campaign hit 11.1%, the control 14.1%, and other half of our testing segment only 10.6%. But! Something else stood out, too.

The control group stalled at a 7% open rate, slightly above my not-quite-Nate-Silver estimate. But the experimental group (again, none of these folks had opened MonkeyWrench in over a year) kept rising to more than 20%. It was 3 times the other inactives, and nearly half the rate of the regular list.

I wasn’t sure how this could be happening, until I remembered how the emails looked in my iPhone’s inbox. There it was, the inbox’s bridesmaid, quietly gray beside the Times Square neon of the subject line: the preheader.

We had changed the preheader when we built the new template. In the control email, the preheader listed a few of the email’s topics, separated by commas: "Subject line data, lots of popsicles, new integrations, and one monkey butt." In the experiment email, we pulled a single quote from within the text: "Max Temkin believes that ‘sending an email is one of the worst things you can do to a person.'" From a single campaign, we can’t say for sure why the Temkin quote got so many more clicks. Maybe it was because the text was intriguing and people wanted to see how email could be so horrible. Maybe it was because the format was different—a sentence instead of a quick series of seemingly unrelated phrases. Right now we can’t say for sure, but we want to learn more.


While we’ve tested subject lines and images and optimal send time before, the results of what I’m now calling "the Hillstrom experiment" were an unexpected surprise. I don’t expect preview text to push opens to 13% in every campaign, but we’re excited to learn more through additional trials. And by finding the right content, we should be able to reactivate more subscribers than we originally thought.

If you’d like to join the MonkeyWrench list, here’s our signup form. Go crazy.