Jun 19, 2012

Measuring Transactional Email Performance with Mandrill

MailChimp recently launched Mandrill, its new transactional email product, and I stumbled across some interesting results while dogfooding it with a couple of side-projects. For those of you who don’t know, transactional emails are just one-off emails triggered by some action like signing up for an account or requesting a password reset.

BillMinder and DebtMinder

BillMinder and DebtMinder

Some back story: by day, Drew and I work in MailChimp’s Mobile Lab. In our spare time, we run a little company called R7, and we make a few financial related iPhone apps (BillMinder, and its lesser-known sibling, DebtMinder). Whenever new users downloaded those apps and created accounts, we used Mandrill to send their welcome emails. This post is about how we used Mandrill’s beautiful reporting tools, as well as some design tweaks to our welcome email, to increase unit sales of one of the apps by 57%.

Our iPhone apps are simple financial tools that work together and feature an optional account that enables one to sync data to their other devices. BillMinder had decent sales going into our little experiment and DebtMinder had poor sales. If you had installed both apps, you got some neat, collaborative features. The trouble was few people knew DebtMinder existed and even fewer people knew about the helpful extras they gained when using the apps together.

Simple Setup

For simpler sending, Mandrill can behave like a normal mail server (SMTP). That’s how we decided to send our emails, although there’s a powerful API available too. It can even handle inbound email processing (if your app accepts incoming email from users, you can analyze it and take actions with it).

Tracking Opens and Clicks in Transactional Email

Open and Click Tracking Options in Mandrill

Open and Click Tracking Options in Mandrill

One obvious way to improve sales was to simply tell customers who purchased BillMinder about DebtMinder. The natural place for this was in our welcome email. But first we needed to get a grip on our current open and click rates, which was easy-peasy with Mandrill’s tracking features. We literally checked a box and chose an option from a drop-down menu to enable open and click tracking in our welcome email. Done.

I know that sounds like a painfully obvious thing to do, but you’d be surprised. A lot of people don’t realize that you can (or should) track the performance of your transactional emails. We’ve used two other transactional services before Mandrill. One didn’t seem to have open or click tracking data, and the other didn’t give much detail. For example, they could tell us what the open and click rate was for an email, but Mandrill will actually give you specifics about who opened and clicked our transactional emails. I’m being told by someone (his name rhymes with "Chen Bestcut") to say that this kind of functional detail stems from our background in email marketing analytics.

Tagging Your Transactional Email

Another thing we did was tag the message by setting a header ("X-MC-Tags"), in this case, "welcome." This automatically created a report for our welcome email so we could look at open and click stats independent of the rest of our emails (like password reset), which would otherwise have skewed the results. You can do a lot more with SMTP Headers in Mandrill, such as setting up open and click tracking per message, as well as configuring Google Analytics.

In hopes of improving sales, we decided to change our email to tell people about DebtMinder and the neat features they’d get if they purchased it. We have less data for the period before the change because we started using Mandrill on April 2.

Mandrill told us that before our design change, the welcome email’s open rate was 45.3% and its click rate was 6%. Not too shabby, but we thought we could do better.

Before Welcome Email Change

Before Welcome Email Change

Before and After

Our original welcome email was pretty simple. It thanked the user for buying the app and that was that.

Original Welcome Email

Original Welcome Email

We first made the email easier to read by improving the layout and increasing the font size. We then added a sentence pointing the user to DebtMinder while explaining one of the features they could take advantage of if they purchased it. Being the social butterflies that we are, we added social media links too. It ended up looking like this:

Improved Welcome Email

Improved Welcome Email

After sending for a while, we checked the reports and were pleased to see that our open rate had jumped by more than 11 percentage points to 57.2%. And the click rate had gone up from 6% to 10.3%, a significant gain of 4.3%. Success!

After Welcome Email Change

After Welcome Email Change

As a member of the MailChimp Mobile Lab, I’d be remiss if I didn’t mention the fact that Mandrill has a mobile app, where we constantly checked our stats:

Mandrill for iPhone (available for Android)

Mandrill for iPhone (available for Android)

All in all, some great reporting and minor changes increased unit sales of our little side-project iPhone app by 57%. Given the wild success that most indie iOS developers enjoy, you can rest assured that we’ll put our newly found lunch money to good use.