Conferences allow your competitors to snoop

We used NDAs to counteract this. As an unintended side-effect, they made the testing seem more exclusive and special, so participants were eager to sign them.

In 2013, we switched from paper to digital forms, via DocuSign. From a logistical standpoint, this was a great move. We didn’t have to keep track of loose stacks of paper after the conference. On the other hand, the signing workflow was rather cumbersome. People had to sign their initials three times and click multiple times to complete the NDA.

Conferences are a great way to build user empathy

Ultimately, user testing is about people, not testing. Both years, we recruited volunteers from non-UX departments within the company: engineering, product, marketing, and sales. It was great to give these people an opportunity to engage with our users over real problems.

And it goes both ways. People love to talk about their job, their pain points, and how your product or service falls short of easing that pain. No, anecdotal data isn’t terribly useful in a design context, but it can help you build a mental model of real-world problems.

CONFERENCES + USER TESTING IS A SCARY COMBINATION

As I mentioned, we recruited volunteers from non-UX teams. Many of those volunteers had never conducted user tests before. It was a nerve-wracking experience for many of them.

In 2013, we instituted a training process to get our volunteers up to speed more quickly. To do this, we instituted a series of training meetings.

In the first meeting, we got everyone in the same room and talked through the testing process and the tests at a high level. Next, we broke up into small groups of two or three people apiece. In these groups, we had volunteers practice facilitating the tests with each other. The test author attended these as well, to spot areas in need of improvement or clarification.

If our volunteers were still nervous about the prospect of user testing, we met with them personally. In some cases, we convinced them to push forward and run user tests anyway. In other cases, we moved them to a less demanding role, usually the role of a greeter.

Conferences are a black hole for data

In the first year, one of our three test laptops was mysteriously wiped of data. The second year, two of our laptops were stolen. We lost all of the test recordings on those machines.

The silver lining was the post-test analysis we did in 2013. Because our facilitators took such rigorous notes, and saved those notes to the cloud, we retained the data, even though the actual recordings were lost.

Process is king, but organization is queen

Keeping things digital as much as possible helps. If you must use paper, don’t use manilla folders. Instead, use three-ring binders with labels to keep your papers collated.

On the digital side of things, consider having a single folder where all conference-related documents and data live. Use tools like Dropbox or Box to keep everything synchronized across machines. Having local copies is critical, in case the network goes down, which it probably will.

Use retrospectives to learn and improve

After the conference, hold a meeting with the core testing team. For the first five or ten minutes, write ideas on sticky notes. These ideas should take the form of things to stop doing, keep doing, or try doing. Put these stickies on a whiteboard, under the appropriate column (keep, stop, or try).

Once everyone runs out of ideas, pick a volunteer. This person groups the stickies by theme (e.g. “communication”, “process”, “people”). Ideally, everything boils down to three to five groups. For each group, find an actionable way to improve that area, then assign each action item to a member of the group. It becomes that person’s responsibility to own it.