Dec 12, 2008

Ethnographic User Testing and Chicken Salad

So I got a call from our accountant the other day. She told me she was having a few issues with MailChimp, and she wanted my help. I told her, "Stop everything you’re doing. We’ll buy you and your entire staff lunch if you’ll let us swing by your office and watch you build your email campaign." She agreed, asked her team what they wanted to eat, and sent me the order.

I debated whether or not to just take them all out to a restaurant. Or to give them a big gift certificate, so they can just go themselves some other time.

But I thought it would be neat to watch them eat their lunch with one hand, and try to build an email campaign in MailChimp with the other. Aarron Walter, our user interface guru, calls this ethnographic user testing. Watching users "in the wild." Dan, my co-founder, calls it "beer can usability." Watch real people trying to use your app with a beer in one hand, slightly inebriated.

User testing "IRL" is hard to do, but you find out all kinds of weird stuff about your product when you do…

For example, here’s one thing it uncovered.

We all know that you should never, ever, ever, copy-paste content from a Microsoft Word document into MailChimp’s (or any) WYSIWYG, right? Because Microsoft adds so much junk to the underlying code? And it screws up everything?

No, we don’t all know that. People are constantly copy-pasting from Microsoft Word. We don’t blame them, because it’s natural. But it breaks every time (for the record, Microsoft calls all that junk "roundtripping code" so that your content and formatting can theoretically be re-purposed across Word, Excel, Powerpoint, etc.

Anyway, people copy-paste from Word all the time, and will never stop doing it. And when it breaks, they of course blame it on MailChimp. Especially if they’re in a hurry to get an email campaign sent. We tell them not to copy-paste from Microsoft Word. If anything, paste it from Word to a text editor like Notepad, so the crazy Microsoft formatting code is cleaned. It’s futile. People just use Word.

Here’s the weird thing.

Sometimes, people see a pop-up window in MailChimp that offers to "clean their Microsoft code from all the junk."

People have thanked us for it. Some people tell us it sucks, because it never truly cleans all the code. These people demand that we fix the "MailChimp Word Cleaner."

But honestly, we had no friggin’ clue what they’re all talking about. We’ve never built such a tool.

Apparently, it’s something built into the WYSIWYG editor that we’re using. It’s a handy tool, but we’ve never seen it in action.

We’ve tried copy-pasting Word content all day, and it never popped up this "Word Cleaner" window. We tried different browsers, different operating systems, different versions of Word, different types and amounts of content. Nothing. We could never replicate what our users were reporting.

Aarron called it the "Mythical Unicorn Feature" that we could never catch.

One-handed operation

But today, while doing our on-site usability testing, our accountant was eating her asian chicken salad with one hand, and building an email campaign with the other:

That’s not our accountant. I just felt like using that stock photo, because it’s so ridiculous looking.

So instead of using keyboard shortcuts like CTRL+C (for copy) then CTRL+V (for paste), she was using the mouse’s right click button for everything.

We observed this:

Right-click –> Copy, then right-click –> Paste.


We found the mythical unicorn!

Right-clicking is what triggered the Microsoft Word cleaner!

All this time, we were trying to replicate the problem by using keyboard shortcuts. No wonder we never found it.

By the way, it does an admirable job of cleaning Microsoft Word code, but it still leaves a few stray bits of junk code behind the scenes. Juuuust enough to screw up your HTML emails. So we STILL don’t recommend copy-pasting from Word.

Too much content?

If you find yourself copy-pasting from Microsoft Word a lot to make your email newsletters, is it because you have tons and tons of content to send?

Should the majority of that content be published on your blog or website instead? And perhaps your email should just be a quick blurb that invites subscribers to click over to your site to read the full article.

Some people just like to write big long articles, but some people just procrastinate. Or they don’t have time to log into MailChimp and build newsletters every single month. So their newsletter content builds up over time.

I can relate to that myself.

Which is why we created our RSS-to-email tool. You publish content and news to your blog. MailChimp takes that content, sticks it inside your HTML email newsletter template (all nice and pretty), then we send it to your subscribers. Automatically.

We discussed this with our accountant. She could setup a blog to post frequent updates. News, opinions on tax regulations, tips for organizing your business records and receipts, etc. And since it’s a blog, she can use it to show her personality. Then, once a quarter she can send the "big official newsletter" to her list, like she currently does in MailChimp. This approach, with the frequent blog updates in between official newsletters, has the added benefit of keeping her list clean and active.

Best of all, she may never have to open up Microsoft Word ever again.


BTW, any MailChimp users in or kinda near Atlanta want a free lunch? Let us stop by your office and watch you work. We’ll hang around afterwards and talk email marketing with you, or show you how to do stuff in MailChimp. Great way to get some free consulting (and food).