Blog

Avatar for bchestnut

When’s a Good Time to Interrupt Customers?

Posted by Ben on


Sometimes, it really sucks being a marketer with a conscience. On the one hand, I want to promote MailChimp’s compelling (imho) new features and benefits to our customers all day long. On the other hand, I know that would annoy the hell out of customers, compelling them to leave our service. So the challenge has always been to find effective-yet-non-annoying-ways to sell stuff to customers.

Um, what about email?

Of course we can send promotional emails to our users. And we sometimes do. But those emails are sent to really small, targeted lists. It’s very tempting to send to our “System Alerts” list of all registered users (that’d be more than 2 million recipients), but we try to reserve that list for the most critical “we moved your cheese” issues. If we send those emails too frequently, people will stop listening.

The Problem with Login Road Blocks

So a few years ago we built an internal tool for making “road block” screens, that tell people about new features as soon as they log in to MailChimp.

If you’ve ever used a web app, you’re familiar with this concept. I think Wufoo probably did this best, with their “since you’ve been gone” screen. It’s informative, it’s small, and it’s quick.

But this roadblock technique never really got used much at MailChimp, because it felt like we were annoying our customers. Unlike with other apps, the stress level is pretty high when it comes to email. So, just like the System Alert emails, roadblocks are now reserved for critical messages (“we changed your navigation” or “it’s a holiday, and support is only partially available”).  Here’s one recent example of a login roadblock:

I’m honestly glad that our road blocks are rarely used, because people are logging in to–you know–get work done. And we’ve learned over the years that MailChimp users tend to procrastinate, so when it’s time to send a campaign they need to do it yesterday. Blocking them at login, popping up advertisements and warnings and doo-dads is just a bad idea.

 

Waiting our turn to speak

But then we got to thinking: what about when users log out? Presumably, they’re all done sending their email, and all the stress and pressure and urgency (and shame of procrastination, in my case) is gone. Maybe this is when our users are most open to suggestion?

So we did a little research. We wondered how many people actually click the “log out” link inside the MailChimp app. We know that a lot of times, people just close their browser tab, thinking that’s good enough (it’s not, btw).

We looked at data for the previous month. Turns out, the majority of users in MailChimp didn’t log out. But among the people who did, we’d get about 750,000 views of the log out screen per month, if we had a log out screen. That was good enough for us to start experimenting.

 

See-Ya-Later

So we built a new screen located at /see-ya-later/ and made it so that clicking “log out” simply redirected people here. This is what our first log out screen looked like:

It pointed users to our “Mobile Friendly Campaigns” landing page. The goal here was to teach our users about all the new responsive, mobile friendly email templates we’ve got in MailChimp. This design resulted in about 1,000-2,000 page views per week for that landing page. Not too shabby. Just for comparison, sending to my massive email list gets more than 500,000 opens and between 10,000-15,000 clicks.

Next, we decided to try promoting Mandrill, our new transactional email product. Here’s a screenshot of the first log out screen we designed for that:

This one featured a screenshot of the app that tried to show that Mandrill’s analytics-heavy, and is very different from the aesthetics of MailChimp (Mandrill’s more for devs, or for techie-marketers). We also wanted to demonstrate that there’s a very nifty mobile app for Mandrill, so there’s an image of an iPhone there. And there’s our witty little take on the startup elevator pitch that we’re all familiar with: “It’s like MailChimp, but for Apps.”

Results were roughly the same as the previous log out promotion: it sent between 1,000 and 2,000 resulting page views to Mandrill.com. Nothing to write home about.

We had a hunch that this screen looked way too similar to the previous logout screen and maybe was suffering from the effects of some form of “banner blindness“ (both were blue, both had smart phones). On that note, we also scrapped plans to launch this third log out screen that was going to promote our MailChimp Mobile app:

More blue? More phones? Not again.

 

So we went back to the drawing board and completely redesigned the Mandrill promotion. This time, we used a bold illustration to emphasize how “Mandrill’s new and Mandrill’s something altogether different from MailChimp” and we traded the MailChimp signature blue for Mandrill’s black:

This one performed way better. It resulted in 19,000 page views on Mandrill.com on the first week, and 14,000 the next week. Here’s a graph of that:

But what about conversions?

For our “mobile friendly” landing page, we didn’t bother tracking conversions, because that was more about educating existing customers. But Mandrill is a brand new app (and users are nice for brand new apps) so we put conversion tracking in place this time.
Signup conversions driven by /see-ya-later/ increased by almost 500%.
App downloads driven by /see-ya-later/ also increased by almost 80%.
Visits to Mandrill.com driven by /see-ya-later/ increased by almost 1000%.
FWIW, the new log out screen was also more effective at generating interest on Twitter:
You can see the first screen’s bump around Sep5, then the next design’s bump at Sep12.
Granted, just about any Twitter traffic at all is bound to look like a spike right now, since overall Twitter conversations are very low for @mandrillapp. But still, this is enough to show that our illustration approach trumps the screenshot approach.

More experiments to come

What’s really nice about our logout screen is that it doesn’t exist on our app servers. When you log out, we redirect you to a simple, static page on our website (/see-ya-later/). So our marketing team isn’t restricted as much by the security requirements of the app (one reason we’ve stopped doing our login screen designs). We can embed videos, tons of screenshots, slideshows or whatever we want here. So long as it’s useful to our customers, of course.
So there you have it. Over the years, we’ve spent a ton of time and effort experimenting with different ways to notify our users about new stuff. We went with our gut and killed techniques that felt annoying to customers (even at the expense of short-term sales and growth). We avoided the easy way out of shotgun-blasting users with intrusive promotions. In the end, it paid off with a really easy way to promote new stuff to our users, in a place that doesn’t get in their way. It’s not as effective as email, but than again, NOTHING IS AS EFFECTIVE AS EMAIL (please repeat that 20 times now).
Experiments with Email Automation
On that note, we’ve also been experimenting a little with email automation as a way to educate customers about our functionality. To give you some perspective, when we started MailChimp 11 years ago we set up a newsletter list. It’s at about 100,000 subscribers now. A few months ago, we created a “Getting Started” series that sends tips every few days to new users. That list is already beyond 500,000 recipients. Needless to say, we’re experimenting a lot with that list. You can see Brad’s lab notes over here. And you can bet that we’ll be launching new features in this area soon (not just features that help you send more, but features that help you send smarter).
Related: