Before we released MailChimp Pro, we tested it. Then we tested it some more. After a year of creating and testing, we were proud to have built something that benefits not only our users, but also the email marketing we share with them. When we started thinking about how to announce the release of MailChimp Pro to the world, we knew we were working with the best tools for the job. And, predictably, we tested even more. Here’s a little look at all the experimentation that went into the creation and early marketing of Pro.
Testing Multivariate Testing
We got warmed up for using Pro to announce Pro by using it to roll out a few more product announcements. Since the MailChimp marketing team is just a 30-second Heely ride away from the product team, they let us try all the new features while they were still being developed. So before Pro was out in the world we started working on 8 versions of an email about our new A/B testing features. We followed that up by testing 5 variations of an email about MailChimp Subscribe.
When it came time to work on our big Pro emails, we were ready. With 2 multivariate campaigns under our belt, we could use what we learned to set up set up a campaign we knew would be a high-performer. There’s a prominent image, immediate call to action, minimal text, and—thanks to our email developer Alex Kelly—a universal design, consistent everywhere from Outlook 2000 to that Android phone where texts scroll across the bezel.
Here’s how the draft of the email changed as we tweaked it:
Comparing Comparative Reports
Our standard reporting features have always been instructive in developing our own email marketing plans, and I knew working with Pro’s new Comparative Reports feature would be even more illuminating. When we first planned Pro’s email marketing, we created 4 sets of messaging targeting 4 types of users. A few revisions later, we settled on a single message for everyone—but we still wanted to break out performance for each group. Comparative Reports made that easy.
Sure, we’ve exported a few lists in our day and are no strangers to Microsoft Excel. But once we generated our first comparative report, I knew there was no turning back. We could do more at once, do it faster, and learn new things that we never expected. Right away, 4 tasks became 1. So why not use all the time we saved to run a few more tests? (Y’all, we love tests.)
We looked at the campaign performance for free users versus paid users, plus the people who installed Pro and some customers we’ve marked as VIPs.
In just a second, we can see how these groups compared to each other, but we’re also able to figure out that 70% of Pro subscribers opened the email and about half of those clicked through to the landing page.
And with additional data passed into our list, we’re able to use Comparative Reports to isolate results for any segment imaginable. Plus, when we re-send our announcement to new susbcribers, we’re able to combine stats from both campaigns with 2 clicks.
I <3 MailChimp Pro
This may sound like tooting our own horn, but really, MailChimp Pro has changed the way I do email marketing for MailChimp. From here on out, every one of our campaigns will use Multivariate Testing, Comparative Reports, or a combination of both. It’s changed how we send, what we send, and how quickly we can iterate. And from what I hear, this is only the beginning. Stay tuned.