Apr 9, 2015
How We’re Testing Timewarp
Part of what I do here at MailChimp is send our MonkeyWrench newsletters, and part of what I do is experiment with how we send emails, so it’s convenient that MonkeyWrench is a great venue for experimentation. It’s an opportunity for us to delight, inform, and show off our users, sure. But it’s also a risk-free sandbox where we can try things out and improve an app that’s used by 7 million people and businesses.
We’ve seen our share of failures, like when we tried replacing text with images to increase clicks. So generally we don’t expect too much out of these experiments. But, much like the TV dinner I ate last night, now and then we’re pleasantly surprised. When that happens, we can take advantage of proven ideas with the rest of our email marketing. Most recently, that happened when I decided to mess around with MailChimp’s Timewarp feature.
Back to basics
For last month’s MonkeyWrench, we focused on the white t-shirt, the artichoke, the most basic of all email statistics: opens.
When you think about why someone decides to read an email, you probably start at the subject line. And it’s totally true that more people will open the "best" subject. But through all of our A/B splits, 3% is the biggest open rate increase we’ve gotten. Not bad, but not huge, either.

I recently rediscovered a series of successful tests using Send Time Optimization and remembered the importance of timing an email so it’s received at the most convenient time for the reader. So I clicked the Schedule button, set a date, and dusted off my old buddy Timewarp. Born in 2010 to 250,000 proud MailChimp users, Timewarp lets you send an email at the same time in every time zone. Schedule a campaign for midnight, and readers in Stockton and Stockholm will have an equal chance to open from a glowing bedside smartphone. Schedule a campaign for 9am, and it’ll arrive just as subscribers are starting their day, whether they’re in Osaka or Omaha. Pretty cool, yeah?
It’s 11:30 o’clock somewhere
So our previous MonkeyWrench earned an open rate of 23.2%. Pretty solid. And assuming that our existing subscribers held the same rate, if all 6,394 new signups opened their first issue, we would’ve reached 26.2%. But even better, this issue was opened by 26.4%—comparatively, an increase of 14%. Like most 10-year-old lists, opens and clicks have decreased over time. Sure, a different subject line could have provided a boost, but we rarely see this much improvement.
We tested again the following week with our vertical newsletters for musicians, nonprofits, and agencies. Instead of relying on comparisons to previous campaigns, we divided each list randomly. One half received their newsletter at 11:30am ET, while a Timewarped replicate sent at 11:30am in each subscriber’s local time zone.

While it lost in our agencies campaign, Timewarp won handily in music and nonprofits against identical subject lines. We gave it one more try with a larger segment when we announced our updated integration with Eventbrite. Through 79,000 deliveries, Timewarp added 165 opens—not quite enough for our product designers to anoint a new default, but enough to make sure we keep an eye on it.
Don’t be a stranger
A wise data scientist once said: "If someone doesn’t want to open, they won’t open, regardless of delivery time."
Your own success with Timewarp will depend on the age of your list, how you’ve collected subscribers, and how they’re distributed geographically. But if you’re a paid user with Timewarp available, you might find a few more opens with a few extra clicks. Give it a whirl!
Bjoern Sjut
hi Brad,
Thanks for sharing. What’s your best practice to divide a list randomly, like you discussed in the blog post?
Because I’m not really aware of a convenient method for that in the interface – we often use approximations (e.g. contains “a” in the “FNAME” and see that we somehow get a good 50%/50% split approximation.
But there must be a better way, right?
04.10.2015
Brad MailChimp
Hi Bjoern,
We’ve definitely used tricks like contains or greater than / less than with name or email address fields. Sometimes this method takes a little wrangling to hit 50/50, but it usually works in a pinch.
For these tests, I opened Excel. First, I exported the lists and left only the email addresses. Then, in the empty second column, I entered =RAND() and generated a random number for each address. (In case you don’t have Excel, Google Sheets has an identical function.) After sorting each row by the highest random number, I copied the top half and pasted the addresses into a new static segment (http://eepurl.com/bhTYxr). The first campaign sent to anyone who was in that static segment and the replicate to anyone who wasn’t.
I actually found this process through a Google search that led me here: http://www.lifehacker.com.au/2012/03/how-to-sort-a-list-randomly-in-excel.
Thanks for reading!
04.10.2015
Bjoern Sjut
Hey Brad,
thanks for the quick reply – my colleague Tanja just told me that also Hairball offers a “give me X% of a list”. Do you happen to know if that’s also a random sample?
Cheers, Bjoern
04.10.2015
Brad MailChimp
Hey Bjoern,
No problem! If you’re using the “give me X%” function in Hairball, that should also output a sample that’s entirely random.
04.10.2015
cory huff
Thanks for the sampling advice. That’s good stuff.
04.15.2015
Lance Sells
Hairball is great but I’ve noticed when I create a segment greater than 750K it times out when uploading. Not sure if it’s just me but it’s happened when I’ve uploaded a big list with 50%.
Aside from that Hairball is highly recommended and much easier than Excel if you have bigger lists.
04.12.2015
Brad MailChimp
Hey Lance,
I just talked with one of our technical leads, and he suggested reaching out to our support team if Hairball times out again. They’ll be able to check your account to make sure you’ve got the best solution for your particular list.
04.13.2015