Should you add conferences to your toolbox?

Having done this a couple of times, it’s clear that there are pros and cons. No user testing tool or technique is a cure-all, and conference-based testing is no exception.

PROS

•Lots of participants Hundreds at a small conference, thousands at a medium conference, tens of thousands at a large conference. Take your pick.

•Easy recruiting Build it and they will come. It helps if you point your laptops into the room, and have the designs clearly displayed on their screens.

•Enables rapid iteration You can easily complete five or six tests in an hour or two. Faster if you have multiple test stations.

CONS

•Chaotic testing environment You know those quiet usability testing rooms with the mirrored glass? You won’t find those at a conference.

•Travel required Unless you’re lucky enough to have a relevant conference in your city, you’ll probably need to fly somewhere. This can be expensive.

•Difficult timing Remember those roadmaps I mentioned earlier? If the design phase doesn’t line up with a conference, find a different way to get the research you need.

In general, this approach works well when you have a predictable product roadmap. If you know what you’re going to be building, and when, you can time the design phase to coincide with one or more conferences.

On the other hand, if you need the flexibility to run tests at a moment’s notice, this approach won’t work well. In that situation, I recommend having a dedicated room for testing at your company, containing all the equipment you’ll need.

Tips to make this work for you

If you’ve read this far and think conference-based user testing is right for you, great! Here are some tips to help you succeed.

•Pick a conference five months in advance You don’t have to know exactly what you’ll be testing, but it’s a good idea to have a target date and venue in mind, so you can start thinking about it.

•Pick a conference with people who don’t know you exist Because we ran testing at our own conferences, everybody knew about us. This self-selection bias prevented us from getting a good cross-sample of our potential market.

•Don’t pick a booth in the busiest hallway As tempting as it might be to get maximum visibility, ask youself if the additional chaos is worth it. In 2013, we picked a booth in a room separated by a half wall from a busy hallway. As a result, we had good visibility without being in the middle of the chaos.

•Don’t write every study yourself The first year, I wrote four of the five user studies. As a result, they were difficult to facilitate and didn’t result in actionable data. It’s time consuming to write a good user test that validates your hypotheses and is easy to facilitate.

•Don’t schedule people in advance When your testing stations fill up, it’s tempting to start a waiting list. Don’t do that. You’ll become beholden to the list and have to turn people away, even when there appear to be empty test stations. Be serendipitous about it.

•Practice running each test on each machine before the conference Murphy’s law. Need I say more?

•Go forth and user test The only thing worse than a poor user testing experience is not doing it at all. If you fail, at least you’ll learn how to do it better next time. If you don’t do anything, you’ve learned nothing.

And that’s it. If you have any questions, please get in touch through Twitter. Thank you for reading.