We’ve been working on some long overdue upgrades to one of MailChimp’s more popular features. While sifting through all the feedback emails we’ve received from customers over the years, we noticed that most of the requests made perfect sense for our audience, but some of the requests seemed strange (and those are the ones that pique my interest the most). We needed a little clarity, so we decided to send a survey to our customers. But surveys can be really intrusive. You’re asking a customer, who’s already giving you money, to give you something more precious: their time.

So here’s how we used list segmentation to maximize response, while minimizing annoyance.

Our customer list is over 2 million. It was pretty damn tempting to send the survey to that entire list, to be honest with you.

That sure would maximize survey response, wouldn’t it? But that would also maximize the bad kind of response (unsubscribes and spam complaints).

Plus, this survey was about a feature (Autoresponders) that’s not used by everyone. Fortunately, we track feature usage in MailChimp down to the user, and we use our own API to create static segments.

So I can build a segment of “only those who’ve actually used Autoresponders:"

There.

Philosophical question: does segmenting my list suddenly make the email more relevant to these 42,079 recipients? Nah. If I had sent the email to my entire list, these same 42,079 customers who’ve used Autoresponders would’ve still received it, and its relevance would be exactly the same to them. Segmentation doesn’t help me increase relevance for these customers so much as it helps me decrease irrelevance for all the other customers.

That’s a really important distinction, imho. Use segmentation to reduce irrelevance, not to increase relevance.

It’s an important distinction because with that in mind, we can reduce irrelevance even more by removing people who aren’t that engaged with us:

Above, I’m creating a segment of my list that has used Autoresponders, and is reasonably engaged with me. Now we’re down to roughly 16,000 recipients. That’s plenty for a survey, and not so much that I piss off huge swaths of my customer base.

But we’re only getting started here…

 

Crafting a non-annoying email

I was in a hurry, so the first thing I did was grab a pre-designed email template, then I plopped in some quick text along with a link to my survey. Then I stopped myself. Just because I’m in a hurry, and my team really, really needs this feedback soon, and even though it’s “just a quick survey” I need to slow down and respect my recipients’ time here.

So I took a break and thought about what would make me actually stop my busy day and take someone’s survey. After about 2 hours of researching my inbox for surveys I’ve ever taken, I came to the conclusion that almost nothing would convince me to take a survey. I mean, it would have to be about a product that I love and use a lot, and the company president would have to write me a nice, personal note about how they’re about to change that product I love so dearly, and that this is my chance to provide some input.

So that’s what I did.

The Subject Line: “Autoresponder Changes Coming”

Here’s the email:

 

Let’s break it down:

1. The subject line is straightforward, honest, and hints at a little bit of urgency. Not the mysterious "three-red-exclamation-point-high-priority-open-to-find-out" kind of urgency, and not the "Final notice! Act now!" kind, but a more subtle kind. Changes are coming. If you’re interested, read this. If you don’t really care about changes to this feature, no biggie. (more tips for writing and researching subject lines)
2. The tiny pre-header text at the top is what’s seen in the inbox, and gives a hint at what’s inside the email. I kind of think it’s the continuation of the subject line: "Changes coming, we could use your feedback." It’s written with a slightly casual tone, asking for help.
3. I threw away the email template, and went with a basic layout.
4. A big headline reinforces what this is all about. I almost didn’t do the headline, because personal notes don’t typically include them. I could’ve used a more raw, plain-text style for the entire email. But I think it’s futile to pretend a mass email is a personal note. People are smarter than that. I can write it in a personal tone, while remaining honest to the medium here.
5. I chose these words: “we’re about to open the hood” and “it’s hard to sort them out” because I want to write like I talk. I could’ve used more official sounding corporate-speak, like “We are planning our next release, so please submit this survey with your feedback and…” But nobody likes that. I’m just a guy in the parking lot with the hood open, and I could use a jump here. Got a sec? Write like you talk. Mostly, you talk using contractions. I’m hitting the send button, but I’m not the one in control here.
6. Post Script: Another thing people (not corporations) tend to do in their notes. This is where I actually drop something useful for people who use Autoresponders: A case study about autoresponders.  I’m not suggesting you use post scripts as a gimmick to get clicks; this is about my choice of content placement. I could’ve led with the case study, then later asked for help with the survey. But I’ve never liked the carrot-on-a-stick approach (again–I really like free will). Also, putting something like this in the post script feels unplanned (which it was), and that feels more human, which feels less corporate. I considered re-writing the message with a whole section dedicated to the useful article, but meh. This is how I talk, so I kept it.

The Results

You read this far? Wow. You actually care about writing a little more like a human, I guess. So here’s what that got me:

That 62% open rate looks awesome ("three times my normal open rate! I totally deserve a raise!"). And the 18% click rate is great, compared to my list’s average 1.6%. But this is an email sent to a segment that’s about 125 times smaller than my normal list size.  And I truly believe those 9,386 people would’ve opened anyway if I sent to my entire list of 2M. Again, I’m using segmentation to reduce annoyance, so the stats I really care about are unsubscribes and complaints.

• 45 people unsubscribed (the last time I emailed my 2M customers,  7,863 unsubscribed)
• 5 people reported it as spam (the last time I emailed my 2M customers, 1,168 reported it as spam)

Note: Over 2,000 people clicked the survey links, but I’m more happy that 13% clicked the tiny post script link. I’m glad they got something useful out of the email. That helps me sleep at night.

After the email was sent, Wufoo emailed me a notification every time someone completed the survey. I received HUNDREDS in the first few minutes. Results are still trickling in today. Here’s a screen from Wufoo’s analytics:

1,305 entries! In the first day alone, I got 847. That’s 1,305 people giving us feedback and helping us prioritize new features and make the product better for thousands of existing and who-knows-how-many potential new customers. Not just that, but a handful of customers are continuing to email me other useful feedback ("Hey, I took that survey a few days ago, and had another thought for you…").

When we compiled our report in Wufoo, we got the clarity we were looking for. Those people suggesting some of the "weird" feature enhancements were people who weren’t exactly representatives of the majority. They were consultants, or people inside very large marketing and sales teams, working at huge, thousand-employee corporations:

Their needs are very different from our average customer. Not in a bad way. Just in a "we’ll revisit that later" kinda way. It was nice to put a face on their requests ("ah, so that’s what the large, B2B-marketers-who-also-work-with-sales-teams want") because had we lumped their features in with the others’, our product would’ve had an identity crisis.

Crisis averted. With that in mind, I’d say losing 50 subscribers in exchange for 1,305 responses was worth it.

 

Changing With Your Audience

Why all this fuss about "minimizing annoyance?" MailChimp’s customer list started off as a bunch of early adopters (who else would sign up for something with an ape for its mascot?). They were tech savvy, and very interested in anything we did to improve our app. Sending to that list used to be so easy. I’d talk about new features, drop in a monkey joke or two, hit send, and bam–big fat open rates. Over the years, as MailChimp became more popular, our list grew into the millions, and the audience became much more broad. Sending to that list got hard. No. Sending to that list got nerve-racking. Open and click rates were declining, and complaints and unsubs picked up dramatically. So around July 13th of this year, we stopped sending to that big list, and started segmenting to small segments of it. Here’s what happened:

 

If you’re sending lots of big "batch and blast" campaigns, and you find your open and click rates declining, it might just be that your list has grown really big (congrats), and now you have clusters of people with different interests. Your audience changed, so you’ll need to change how you talk to them. That’s not a problem to be taken lightly. It’s way bigger than an email problem. Lots of companies fail when they’re unable to detect shifts in their audience, and then adapt their marketing strategy accordingly. But all I can give is email advice. So before you hit the send button, I suggest you ask yourself: "Is the email that I’m sending useful for all my recipients?" If the answer is no, try segmenting the list down and ask yourself that question again. Keep segmenting and whittling down until the answer is yes.

 

P.S.

This is somewhat related: The Elegant Email, by Michael Lopp. Much has been written about writing more concise, manageable emails out of respect for your recipient ("keep it down to three sentences!"). This post, like all the others, gets a little too formulaic for me, but lines like "Say something of substance with your subject. (Perhaps with poetry.)" and "A Sense of Doneness and Humanity" emphasize the trait that separates a great permission marketer from an average one: restraint. Sometimes, being useful (and remaining relevant) means choosing to send nothing at all.