Attempt #1: puppetconf 2012

Once a year, Puppet Labs hosts PuppetConf, a tech conference for IT professionals. In 2012, it was held at the Mission Bay Conference Center in San Francisco and 750 people attended.

Mission Bay Conference Center at UCSF, the site of our 2012 user testing. (Image: Greentech Media³)

Two of us prepared five studies and set up three user testing stations in a high-traffic hallway. Each user testing station consisted of a laptop, a stack of test scripts and NDAs, and a volunteer to help facilitate the tests. We had about 16 volunteers, and ran 50 user tests.

This was a great experience, but we didn’t get much actionable research out of it. Our focus was on data gathering. We didn’t bother to analyze that data until weeks after the conference, which meant it had gathered dust. In addition, the things we tested weren’t on our product roadmap, so the research wasn’t timely anyway.

Attempt #2: puppetconf 2013

In 2013, we repeated our user testing experiment. That year, it was held in the Fairmont San Francisco hotel and 1,200 people attended.

Five of us prepared six studies and set up three user testing stations in a room adjacent to a high-traffic hallway. We added dedicated lapel mics and three-ring binders to keep our scripts organized. With the same number of volunteers (16), we ran almost twice as many user tests (95).

This year was vastly more successful than the previous year. We pulled analysis into the event itself, so we got actionable data more quickly than before.

Fairmont San Francisco, the site of our 2013 user testing. (Image: Brad Coy⁴)

Unfortunately, we didn’t go the extra step of iterating on what we learned during the conference. Our product wasn’t affected until months later. It was a step in the right direction, but too slow to be considered agile.

What did we learn?

In 2012, we made a large number of mistakes, but we learned from those mistakes, improved our tests and testing process, and doubled both the quality and quantity of the tests in 2013. So, don’t be afraid of failing. A poor user testing experience will only help you learn and improve for next time.

Here are some of my observations from those experiences.

Conferences let you cut the fat out of recruiting

Recruiting is very time-consuming. We have a full-time position on our research team at Puppet for that very purpose. But at conferences, people are already present and willing to engage with you. All you need to do is show up.

In a typical user study, we send out a screener email to 50–100 people in our testing pool. A lot of people won’t respond, and of those who do, only some will meet the requirements for the test. It takes time to get enough valid responses, and sometimes we have to widen the net, which takes more time.

Conferences let you validate your entire roadmap

In both years we had more interest in testing than we could facilitate. In 2013, the 95 participants who tested with us were far more than we needed.

If you decide to conduct self-guided, quantitative usability tests, you can run even more tests. In 2014, our research team had over 200 people take a single usability test.

Conferences are chaotic, but process can help

In 2012, we had a simple four-stage process: greet, recruit, test, and swag.

1. Greet Every time someone came to our booth, we had a greeter volunteer who said Hi and told them what we were doing.

2. Recruit Next, we asked if they wanted to join our Puppet Test Pilot pool for testing opportunities throughout the year. If so, we scanned their badge.

3. Test If we had a test station available, we asked if they wanted to take a 15–20 minute user test. If so, the greeter introduced the participant to a facilitator at one of the stations.

4. Swag At the end of the testing, we thanked each participant, and gave them a limited edition T-shirt and a signed copy of Pro Puppet.

This process worked well, but there were a couple of obvious holes. First, we didn’t have a good screening process, so there was no guarantee that a participant was a good match for the tests. Second, we didn’t have a plan to quickly learn from the tests and act accordingly (see: agile).

To correct these shortcomings, we introduced two additional steps in our 2013 testing process: greet, recruit, screen, test, swag and analyze.

•Screen At the beginning of the testing process, the facilitator asked the participant six questions, one for each user test. If the answer was yes, we knew they’d be a good match for the test.

•Analyze At the end of the testing process, the facilitator filled out a short form. Each user test was allocated a text field, with the study hypotheses alongside. The facilitator entered their notes, and marked the validity of each hypothesis.