Mar 19, 2010

Timewarping your A/B tests takes some getting used to

So I find myself checking my emails a lot after lunch, and then again around 5pm, just before I leave the office I’m supposed to leave the office. And of course that made me want to A/B test delivery time for my March MonkeyWrench newsletter at 1pm vs. 4:30pm.

In the past, doing an A/B test on "delivery time" wasn’t totally accurate, because it wasn’t really true to each recipient’s timezone. If I send at 1pm ET, that’s actually 10am PT (check my math there).

But with our new Timewarp feature, we actually deliver based on each recipient’s timezone. So far, here are the results I’m seeing:

timewarp-map

In the report above, you can see that MailChimp has covered most of Asia, and is working its way across the globe, timezone by timezone. Like Santa Claus. But for email. Anyway, sending a Timewarped campaign can be weird, and it takes some preparation…

Plan Ahead and Arrive Early

The first thing you’ll note is that when you schedule a campaign with Timewarp, we require that you schedule it 24 hrs in advance.

timewarp-must-be-in-future

So while I was working on my campaign on Wednesday afternoon, I really wanted it to go out on Thursday at 1pm. But MailChimp wouldn’t let me, because somewhere on the planet, it was already 1pm on Thursday. Sort of. Technically, 24 hours is a lot of padding that we put in for simplicity’s sake—we may refine later.

Furthermore, I made last minute edits to the newsletter on Thursday, which pushed out the earliest delivery window to Friday.

This basically means that if you want to use Timewarp, plan ahead. It’s like getting to the airport 3 hrs early for international flights.

Time Travel May Sting A Little. Don’t Panic.

So I got the thing scheduled for Friday, March 19th, at 1pm. Got it.

Thursday night, I started to notice some auto-replies in my inbox. What the? My campaign was already delivering on Thursday night!?!? But I distinctly remember MailChimp telling me it would go out tomorrow.

In a moment of panic (ok, sheer stupidity) I sent an email to our developers that my Timewarped campaign delivered too soon, and they might want to check for bugs.

Normally, the nerds reply to my tickets really fast. This time, nothing. In fact, the ticket was silently set to "resolved." I’ve learned over the years that basically, this is their polite way of saying "Ben’s having a PHB moment."

Ah. It’s already 1pm. Somewhere.

Interesting Side Effects of Time Travel

This does have one cool side effect. Normally, I get a lot of auto-replies after I send my newsletter. You know, the "I’m on vacation, but will reply as soon as I’m back" kind. Now, my list is so large, I get a lot of auto-replies. So I normally have to filter them into a folder. Later, I go through all those auto-replies to click on all the challenge-response messages, and sometimes, for kicks, I actually reply to auto-replies ("Sooo, how were the Hamptons?"). People get a kick out of knowing I’m actually a human over here, and it sparks some really good customer conversations sometimes. You should try it!

Oh, but anyhoo, Timewarping your email makes those autoreplies a bit more manageable. They come in smaller chunks. Instead of hundreds of them flooding my inbox instantaneously, I’m getting chunks of 10 from Japan, then 10 from Italy, and so on. It’s really neat knowing where Santa — er, MailChimp — is at any moment.

Another side effect, for our company at least, is that whenever I send a big campaign, it usually triggers a huge flood of customer service requests. People get an email newsletter from MailChimp, and they have questions about new features, or they’re reminded of questions they wanted to ask last week, etc. So normally, I have to consider scheduling my sends around when our customer service team is most heavily staffed. Lunchtime is not a very considerate time to send. But delivering in timezone’d chunks seems to help spread out the influx.

Stats Take Time To Generate

Here are my A/B results so far:

already-sent

That’s actually the screenshot I took during my, um, "episode" mentioned earlier. You’ll get a chuckle if you look at the filename.

At first glance, it looks like 1pm is by far the winner. But I’ve got a loooong way to go before Santa’s done sending. On your Timewarp map, you’ll want to take note of the following:

pending-sent

The green numbers under the timezones means "sent" and the blue means "pending." So I’ve got 13k more to go here.

At the top of the time zone map, you’ll see this:

pending-group-b

Notice the link to view each group. This tells me that it won’t finish sending till 11:30pm (my time). So if you typically like to send your campaign stats to your boss, or clients, remember that Timewarped messages will take longer to travel the globe.

Tweets take longer too

I’ve grown accustomed to clicking the "Send" button in MailChimp, then firing up twitter to see how many people talked about it. It’s quite an ego boost to see a dozen or so re-tweets for your campaign (which is why we’ve added so much social tracking to MailChimp).

But again, since the newsletter is being delivered in chunks, the twitter love is spread a little thin. So far, I’ve twacked one immediate re-tweet of my campaign’s "auto-tweet" (explanation here). Then one tweet from somewhere in the Netherlands about a link from my newsletter that pointed to this Deliverability guide.

Looks like it’ll be a while before the campaign hits the bulk of my subscribers, and the tweets show up. Timewarping does require more patience than normal campaigns. But then again, if I sent the email "normally" on Friday at 4:30pm, there’d be a lot of people in Asia getting it on Saturday. For them, it’s probably nicer that they’re getting it on a workday. Hmm. I guess it’s all relative!

800px-Albert_Einstein_1979_USSR_Stamp