What is user testing?
Let me start by defining what user testing is, and what it is not.
User testing is…
•Formal Your goal is to get qualitative feedback on a single design iteration from multiple participants. By keeping the sessions identical (or as similar to one another as possible), you’ll be able to suss out the commonalities between them.
•Observational Users don’t know what they need. Asking them what they want is rarely a winning strategy. Instead, you’re better off being a silent observer. Give them an interactive design and watch them perform real tasks with it.
•Experimental At the core of any user study is a small set of three to five design hypotheses. The goal of your study is to validate or invalidate those hypotheses. The next iteration of the design will change accordingly.
User testing is not…
•Ad-hoc Don’t accept what a single person says at face value. Until you get signal from several people that a design is flawed, withhold judgment. Once five or six participants have given consistent feedback, change the design.
•Interrogative Interviews are useful for learning about users, their roles, and their experiences. But keep it brief. Interviews tend to put the focus on what people say they do, not what they actually do.
•Quantitative Because the sample size is small, you can’t make strong statistical extrapolations based on numbers alone. If you care about numbers, look into surveys, telemetry, and self-guided usability tests instead.
What is a user study?
A user study is a research project. It starts with a small set of design questions. You take those questions, reformulate them as hypotheses, devise a plan for validating the hypotheses, and conduct five or six user tests. Once done, you summarize the results and decide on next steps. If the findings were clear, you might make improvements to the design. If the findings were unclear, you might conduct an additional study.
You
won’t get it right the first time. Test your design, iterate, and
repeat. (Image: raneko1)
A good user study has clear and measurable outcomes
If you have clear expectations, it will be much easier to take action on what you learn. This is often accomplished with hypotheses: testable statements you assume to be true for the purposes of validation. Examples of good hypotheses include:
•Users can add an item to their shopping cart and check out within five minutes.
•Users want to click on server-related error messages to see additional details.
•Users are not frustrated by the lack of a dashboard in the product.
A good user study is easy to facilitate
This is especially important if you are not the facilitator. If the facilitator is inexperienced with user testing, you’ll need to provide a test script which is easy to understand, keeps the test on track, and explains what you are trying to learn from the test.
A good user study must be sufficiently detailed and interactive
If you want to measure a user’s reaction to an on-screen animation, you probably need a coded prototype. If you need to decide whether a particular screen can be omitted from the final design, a set of PSD mockups will do. Needless to say, this is a lot of moving pieces. Effective user studies are rigorous, and rather expensive to pull off as a result. If you cut corners, you may second-guess your results and need to run another study to be sure.