Feb 17, 2010

Why did my open rates change?

What’s a good, average open rate? Our customers ask us that question all the time. So much so, that a few years ago we analyzed close to 300 million emails and posted our findings to ChimpCharts. Next, we embedded that data right into your campaign stats:

learn more about industry comparisons

learn more about industry comparisons

But I send a lot of campaigns. Because I use MailChimp’s RSS-to-email tool wired up to this blog, I send almost daily. And I look at my stats all the time. So I already know my average open rate, and I already know my click rate, and I already know that I’m usually a few percentage points above industry average (c’mon, step it up a little Internet & Software industry!).

Nowadays, I find myself seeking anomalies in my stats instead…

In other words, I just want to know if something’s changed about my performance.

So my new favorite campaign stat is MailChimp’s performance advice:


MailChimp gives a little advice

You may have noticed that little link inside your campaign reports. Click it, and MailChimp will tell you (with little red or green arrows) if your stats are trending up or down.

In the screenshot above, it’s telling me that my open and click rate are lower than normal for me. That would normally make me wonder if my content triggered some spam filters. In which case, there’s not a lot you can do with RSS-to-email. I’m not going to worry about how spam filters think while writing for my blog. But if it were a normal campaign, and I didn’t do this before sending, I might run it through our inbox inspector or delivery doctor tools to uncover any problems.

I tend to not worry so much about my content, though. I’m on a dedicated IP address (I send a lot, and my team uses my account as a guinea pig for stuff), which means I tend to look for infrastructure issues more than content triggers.

So I look at my domain performance report to see if I’m getting blocked by any particular ISP:


In particular, I’m looking to see if there are an inordinate number of bounces from any single domain. Nothing here.


I go back to my performance advice, and notice this little link advising me to go download my master campaign report (hey MailChimp team, didn’t I ask to name this our B.A.R.F. report?):


So I download that big giant spreadsheet, which contains ALL my stats across ALL my campaigns ever sent.

This way, I can slice and dice numbers any way I want, and build my own custom graphs (see how you can use this report to calculate your best day to send).

I sort my columns so that I can focus on my daily blog updates, and immediately, I notice my open rate declining in November:


Whoah. I was in the mid-30’s there for a while, then bam—mid 20’s. Did my content start sucking?

No, because it always sucked.

If you look at the 3rd column in that screenshot, that’s my list size. You’ll notice that something happened between November 5th and November 7th that grew my list by around 50%. Yowzers.

Could it have been some spike in my website traffic? Some amazing blog post that I wrote that got the world excited about email marketing, and so they stormed over to my blog and signed up to hear more from yours truly?

That’s when I remembered that we introduced a new report called Site Analytics360 a while back (note to self: blog about Site Analytics360). It takes your Google Analytics stats, and overlays your MailChimp and twitter stats, so you can see how your email campaigns are driving traffic to your website, and spot trends or anomalies along the way. You can find it under the Reports tab in MailChimp.

Just look for this button:


So here’s a screenshot of my blog stats, overlaid with my MailChimp stats.

Nope. No spike in traffic to the site, and definitely no spike due to my blog content. Shucks.


So what was it? Why’d my daily updates list grow 50%, thereby reducing my opens and clicks below my usual average?

Ah. I remember now.

Some time in early November, I imported 300 seedlist addresses to track my deliverability through ReturnPath.

So I’ve basically got a bunch of "fake" subscribers on my list, weighing down my engagement levels.

I go to my list and search for "returnpath" and sure enough, I see subscribers like this:


who were all imported on November 5th, 2009. It’s all coming back to me now.

By the way, if you import any seed lists, you might want to "tag" them in some way. I gave them all sequential ReturnPath1,2,3… FNAMES, so I can easily mass unsubscribe them later.

So now I’ve solved the mystery of my declining open rates. Feels good to know why it happened. Hopefully, if you ever get a client or boss breathing down your neck about declining email performance, this tutorial can help you uncover any problems.