Nov 25, 2013

Subscriber Engagement Over Time

As a data scientist at MailChimp, I get to dive into some pretty hefty datasets to search for patterns and learn more about our customers. Most recently, we set out to learn how a subscriber’s engagement behavior changes over time. We’d previously covered this in a short blog post two years ago that showcased the work of Dan Zarrella. I figured I’d pop in, update a few coefficients, and get back to other projects I was working on. Boy, was I wrong. We’ve grown a lot in the past two years, so there’s a lot more data to be analyzed.

We sifted through over 45,361,424,068 sends, 1,286,970,135 clicks, and 1,250,527,700 list subscriptions over the last year to find out how subscribers’ clicks evolved over time.


In order to analyze how a reader’s engagement evolves over time, you can’t look at aggregate open and click percentages for users. Each user’s list is like a waterfall, evolving over time as new subscriptions and unsubscribes come in (hopefully more of the former!). To do this right, then, I had to look at the data on the subscriber level and analyze each individual subscriber’s story. When did they subscribe? What did they receive? What did they click on? When did their engagement die out?

Some issues come up when comparing lots of subscribers against one another: They don’t get campaigns at the same rate. Some are longtime subscribers, and others are just starting out on particular lists. They’re members of different list segments.

I got around these difficulties by looking at subscribers with respect to when they signed up, and numbered each of the campaigns they got. That way, you can see how their click rate changed with respect to their individual subscription date in a normalized way. By using this approach of counting "sends since subscription date," I calculated the click rate for each sequential campaign. All the clicks and sends for every subscriber who got a first campaign were combined into a "first send click rate," same with a second campaign, third, and so on.

Now let’s take a look at the results.

High-level engagement degradation

At a high level, we see that click rate and email engagement half life hasn’t changed much in the two years since our previous analysis. Overall click rate starts just above 5% on the first campaign and declines over time, approaching 3.5% by the 100th campaign for the users who make it that far.

Aggregate Plot



But that’s just a high level view. What about other factors, like how daily deals might compare to e-commerce and traditional retail? Segmenting by the user’s industry, we can see some clear differences.

Segmenting by industry

In the graph below, engagement over sends is broken out by industry. Here, you’ll see media and publishing, e-commerce, retail, and daily deals.

Industry Plot


When someone subscribes to a tech blog newsletter or a literary journal’s newsletter, that indicates that the subscriber wants to read the content, and will probably continue to read the content going forward. This is different than many of the subscription practices used in retail and e-commerce, where the newsletter, even while opt-in, often comes as an add-on to a purchase. And this difference in subscription context and expectation is born out in comparing publishing engagement degradation versus that of retail and e-commerce users. Publishing can maintain a reader’s interest for longer. So it makes sense that if a brand publishes content first and foremost, readers will be more engaged over time.

Diving into daily deals vs. e-commerce and retail, daily deal senders start at the same engagement level that e-commerce and retail senders have on their 20th campaign. That’s quite a disadvantage, but it’s not necessarily surprising—the variety and frequency of offerings from daily deals senders drive disengagement. Over time, though, e-commerce and retail decrease to match daily deals in terms of performance.

Segmenting by recipient domain

Taking a look at receiving domain (across the top 3 domains: Gmail, Yahoo, and Hotmail), you’ll see that there’s definitely a difference between them.

Domain Plot


Gmail subscribers start off more engaged compared to Yahoo and Hotmail subscribers, but over time, they too seem to lose interest and stop clicking at the same rate.


This final graph is my favorite, since it highlights a common theme that comes up around permission-based marketing. Not all senders use double opt-in subscription practices, but it’s clear that when a recipient double opts into a list, they’re giving a big thumbs up to the content. Initial engagement among double opt-in email addresses is double that of single opt-in addresses.

Opt-in Plot


Wrapping up

Any way you slice it, the engagement of an email address on your list will likely decline over time. Unless the subscriber is your mother. People’s interests and needs change over time. But the starting place for a reader’s engagement, and the pace of that disengagement, are controlled by the sender’s content and sending practices.

What kind of content do your subscribers want from you? If you’re just starting out, think about how you can use email to build a community. Show people behind the scenes, and share what you’re learning along the way. If subscriber engagement is decreasing early on, try front-loading the experience with useful autoresponders. This research shows us how important it is to make the most of each subscriber’s limited time on your list.