Nov 5, 2012

Faster, More Powerful List Searching

We just added a subscriber search box to the Dashboard:

Sounds kinda simple, but it was a pretty serious undertaking behind the scenes. See, before making search so easily accessible, we wanted to really beef up our underlying search service to make it faster.  And that’s not an easy task when you consider the fact that we have a ton of small users with small subscriber lists (in the hundreds), and a bunch of large users with ginormous lists (in the millions). The fact that we built MailChimp to scale along with customers as they grow from startup to gigantic corporations has always been our most difficult challenge, but also our best differentiator. Would’ve been a lot easier to choose a database architecture if we just focused on small business or enterprise. Building the app to serve both (plus everything in between) makes things more–interesting. Anyway, in the end we had to build an entirely new search service, and then move 500GB of raw text data, 2 billion records, and 1TB of clustered, indexed search data.

So what can you do with all that power? Well let’s say I want to find all the people named "Ben" across all my lists, because I want to tell them about BenConf, a tech event for people named Ben (I’m serious).

I can just type "ben" in the Dashboard, and BAM:

* Previously, a search like the one above only worked on small lists, and only scanned 3 fields in your lists (fname, lname, and email).

 

Searches All Fields

But the new search tool doesn’t just scan names and email addresses, like the previous one did. It performs full text searching across all 30 fields in each member profile. So I can type something like "Germophobe" and it finds that word in the Notes field of my profile:

 


New Notes Field

Oh, that reminds me: we added a NOTES field to all member profiles. 

To add or edit notes for any member profile, just click the edit button:

 

Then type in your notes:

This is just a first pass. You’ll eventually see Notes available via the API, and popping up in our mobile apps (especially apps like Golden Monkeys or Chimpadeedoo), and ideally working in conjunction with our various CRM integrations out there.

 

Search Tip: If I search for Ben Chestnut like this:

It returns all entries with "ben" and all entries with "chestnut" in them.

So I get people named Bennet, or Benson or Benoit, and other people named "Angela" who work at companies like "National Benefit" or "Chestnut Inc." Perfectly fine people, I’m sure.

But putting quotes around "ben chestnut" like this:

Only returns subscribers that have ben AND chestnut together. Where they belong.

 

Add Subscribers More Easily

On the Dashboard, you’ll also note that there’s a new "Add" button, so you can quickly add subscribers without having to go all the way to the Lists screen:

 

These tweaks are especially handy for those occasions when you need to do some quick manual entry of new subscribers, or when you need to just find someone on a list to edit their status. Log in to the Dashboard, get your work done, and log out.

 

Still Indexing

The new search tool is live, but is still indexing data from last Friday. So you might not have the most recent 3 days of subscriber data available in that search box yet. Just to give you an idea of the scale of this project, the last 3 days of data produced by our 2.5 million users amounts to roughly 30,000 separate files, each with 2,000 lines of code in it. This is because we’re not just indexing your subscriber records, but also campaign activity–so that we can make some of the research from our Email Genome Project actionable within your account (subscribe for EGP news here, if you’re interested in learning more). Exciting times. We anticipate those 3 days of data will be fully caught up and available tomorrow.