Apr 9, 2015

How We’re Testing Timewarp

Part of what I do here at MailChimp is send our MonkeyWrench newsletters, and part of what I do is experiment with how we send emails, so it’s convenient that MonkeyWrench is a great venue for experimentation. It’s an opportunity for us to delight, inform, and show off our users, sure. But it’s also a risk-free sandbox where we can try things out and improve an app that’s used by 7 million people and businesses.

We’ve seen our share of failures, like when we tried replacing text with images to increase clicks. So generally we don’t expect too much out of these experiments. But, much like the TV dinner I ate last night, now and then we’re pleasantly surprised. When that happens, we can take advantage of proven ideas with the rest of our email marketing. Most recently, that happened when I decided to mess around with MailChimp’s Timewarp feature.

Back to basics

For last month’s MonkeyWrench, we focused on the white t-shirt, the artichoke, the most basic of all email statistics: opens.

When you think about why someone decides to read an email, you probably start at the subject line. And it’s totally true that more people will open the "best" subject. But through all of our A/B splits, 3% is the biggest open rate increase we’ve gotten. Not bad, but not huge, either.

timewarp_ab

I recently rediscovered a series of successful tests using Send Time Optimization and remembered the importance of timing an email so it’s received at the most convenient time for the reader. So I clicked the Schedule button, set a date, and dusted off my old buddy TimewarpBorn in 2010 to 250,000 proud MailChimp users, Timewarp lets you send an email at the same time in every time zone. Schedule a campaign for midnight, and readers in Stockton and Stockholm will have an equal chance to open from a glowing bedside smartphone. Schedule a campaign for 9am, and it’ll arrive just as subscribers are starting their day, whether they’re in Osaka or Omaha. Pretty cool, yeah?

It’s 11:30 o’clock somewhere

So our previous MonkeyWrench earned an open rate of 23.2%. Pretty solid. And assuming that our existing subscribers held the same rate, if all 6,394 new signups opened their first issue, we would’ve reached 26.2%. But even better, this issue was opened by 26.4%—comparatively, an increase of 14%. Like most 10-year-old lists, opens and clicks have decreased over time. Sure, a different subject line could have provided a boost, but we rarely see this much improvement.

We tested again the following week with our vertical newsletters for musicians, nonprofits, and agencies. Instead of relying on comparisons to previous campaigns, we divided each list randomly. One half received their newsletter at 11:30am ET, while a Timewarped replicate sent at 11:30am in each subscriber’s local time zone.

timewarp_non

While it lost in our agencies campaign, Timewarp won handily in music and nonprofits against identical subject lines. We gave it one more try with a larger segment when we announced our updated integration with Eventbrite. Through 79,000 deliveries, Timewarp added 165 opens—not quite enough for our product designers to anoint a new default, but enough to make sure we keep an eye on it.

Don’t be a stranger

A wise data scientist once said: "If someone doesn’t want to open, they won’t open, regardless of delivery time."

Your own success with Timewarp will depend on the age of your list, how you’ve collected subscribers, and how they’re distributed geographically. But if you’re a paid user with Timewarp available, you might find a few more opens with a few extra clicks. Give it a whirl!