Jul 21, 2014

The Little QA Team That Could

As of today, MailChimp’s QA team consists of one team leader, one engineer, and four analysts. That may not seem big, but it’s huge compared to where we were last year. Back then, we had just one QA tester—me!—working out the kinks of each of our monthly releases. I’ve learned a lot over the past few years about the ups and downs of being a one-woman shop and about the challenges of managing a team in the midst of a huge (like, 500% huge) growth spurt. Namely, that the ability to move fast and adapt quickly is just as important as knowing when to ask for help.

Started from the bottom

I started at MailChimp in 2009, managed a team of support agents for a year, then became the company’s first full-time QA person in 2012. It was up to me to figure out what QA would look like at MailChimp, and I had to figure it out fast: In my first 9 months on the job, we added nearly 1 million users.

Meanwhile, our devs were shifting into multiple teams and reworking our release workflows, and we were all collaborating to stabilize releases for users. Our developers were more quickly pushing out new features, each more complex than the last. Releases required more time to research users and investigate features. Testing was done in 5-day timeframes, and our work-weeks often extended into work-weekends.

Operating as a one-woman QA shop during this time of crazy growth allowed me to try out new testing methodologies without messing with the ebb and flow of a team. I could make mistakes without fearing backlash and try out new ideas on a whim. Most of the time, this worked well for our rapid testing timelines. The introduction of multi-user accounts, our drag-and-drop editor, and testing 3 new data centers were some of my biggest accomplishments as a solo testing analyst.

This time last year, our investigative testing processes and user-focused, exploratory philosophy were put to the ultimate test as I worked to conquer our redesign, the biggest release in MailChimp history. But as our UX team began releasing previews of the new MailChimp, and as I planned how to approach a full overhaul of our application in web and mobile, I quickly realized the task at hand was way bigger than 1 person. I needed help, pronto!

MailChimp has always relied on small, efficient teams to accomplish our big goals, so I was hesitant to bring on another full-time QA tester. Instead of building a full team, I got help from our API liaison and borrowed a support agent, TK, to help me test every page, every function, and every connection in the new app. A few months into the project, I realized I needed long term help. TK, who had proven himself to be incredibly helpful, soon became QA’s second employee, and I went from sole proprietor to teammate.

Now we’re here

TK and I anticipated that post-redesign life would be more manageable. But our developers still had a lot of ideas bouncing around and more users had come on board, which drove the creation of even more features. Reinvigorated release teams were making bold choices and pushing MailChimp development to a new level. And to match the pace of development, our QA deadlines were still incredibly short. Once again we needed some help, so we decided to bring on another analyst. In 3 months’ time, the size of our QA team had tripled.

With 3 full-time testers, we soon introduced a new week-long testing cycle that gave us space to better plan and prepare for each release. Running as a small team allowed us to pivot with changes over our 5-week development cycle and gave us time to experiment with testing processes I hadn’t been able to implement while flying solo. We could dig deeper into testing, and we were finally able to get a few steps ahead of our stress points. It’s easier to see the big picture and know where you’re going when you have time to reflect on processes and upcoming changes.

In order to implement programmatic solutions to our testing problems, TK was promoted to full-time engineer, which is great because he understands our process and all the problems we’re trying to solve. We’ve also taken on regular mobile testing, API testing, and we’ve been collaborating with more teams across the company, helping the Integrations team test our SalesForce integration and new experts directory. And we’re working on tools to give us more analysis time.

We’ve brought on yet more analysts to fill the gap after TK’s promotion and work on new projects, but our QA team isn’t done growing yet. We’re working to hire another analyst to help with releases and future projects, like testing our Knowledge Base. Looking back, it’s incredible that we were able to do so much with just one tester. I can’t wait to see how much more we can accomplish as a team.