May 21, 2013

How Member Ratings Work

If you ever tried to segment a list by engagement, you may have wondered about our member-activity rating. I’m talking about the little stars that look like restaurant reviews for your subscribers. I’m not sure how five-star restaurants get their rating, but I suspect it’s a little different than email.

To come up with a member rating, MailChimp tracks open and click data and measures that against your sending frequency. Actually, we get a lot of questions about these little stars and how exactly a two star subscriber becomes a five star subscriber. To find out how the sausage is made, we’ll need to look at the code behind member ratings.

Our five star system is based on a twenty point activity score, and we weight those activity scores unevenly. An even distribution might be good for my OCD, but loading activity points around four and five stars ensures that high engagement is easily remembered while low engagement is easily forgotten.

Handling Bounces

Have you noticed that new subscribers start out with two stars when they get added to your list? If so, you may be wondering why our activity score goes down to negative five. Don’t worry, your subscribers can’t earn a negative activity score by ignoring email. We reserve the negative range for addresses that soft bounce, hard bounce, or see spam complaints.

Actually, member rating gives us a cool opportunity. When an email address hard bounces, the ISP is telling us to never try sending to that address again. However, hard bounces are not always valid. If your subscriber has a positive member rating and we see a hard bounce, we’ll reset the activity score to zero instead of automatically removing the address. Cool, right?

Okay, I oversold that. It’s not "cool," but it’s definitely better than the alternative.

Send Frequency?

In order for member ratings to accurately measure engagement, subscribers have to lose points for not engaging with your email. We handle that by decrementing activity scores after you send your campaign. Don’t panic! We don’t decrement these scores every time you send an email. Instead, we factor in your send frequency.

Why does it matter how frequently you send? Well, the law of demand says the quantity demanded and the value of a commodity are inversely related. In this case, your email is a commodity to your subscribers, and regardless of your content, the more frequently you send, the less value any single email has. To put it simply, ignoring a monthly email is a much bigger deal than ignoring a daily email. Here’s the schedule we use when decrementing activity scores…

Send Frequency Decrement Frequency
Once a month or less Every 2 emails
Up to once a week Every 3 emails
Up to twice a week Every 4 emails
More than twice a week Every 5 emails


So there you go. That’s the nitty-gritty on how Member Rating is calculated. Employees must wash hands, and preparation surfaces must be clear of contaminants. Keep in mind, this post is a snapshot of the member rating and activity score algorithms as of the posted date, and it’s entirely possible those algorithms could change going forward. Besides, do you really want to sit down and figure this out? Let MailChimp handle the dirty work.