As product people, one of the very first things we learn is that users are the ultimate judges of our digital product. And we’re not disputing the latter statement. What we’re disputing however is that talking to users is always a credible source of information.
There are a plethora of pitfalls when it comes to user interviews and usability testing. While it’s fair to say that some of the inaccuracies may be caused by designers or product managers, users are people too. And as any people, they’re prone to bias.
In this article we’ll take a look at all of the ways users’ bias can trick you, and what can you do about it.
Let’s dive right in.
TLDR;
We know that some of you are in a hurry, and need insights fast. That’s why we’ve decided to create a more of a visual summary of the contents of this article to deliver more value in a shorter span of time. We hope it helps 😄
If you’re looking to get into this topic a little deeper, we welcome you to keep reading.
1. Users don’t always buy what they thought they would
One of the common questions we ask our users is if they are comfortable with the price point of our product. If the predominant answer is “no”, then you definitely have some room for improvement. However, if the answer is “yes, we’re ready to pay for your product”, it might be too early to celebrate.
Yes, positive comments about your pricing is an indication that users might make that purchase. But this alone isn’t proof. You can find countless examples online of products failing, although user research/usability testing suggested the opposite, and vice versa.
The most common text-book-example of products failing although user-generated input suggested otherwise, is the New Coke, which was a new flavor of Coca-Cola launched in 1985. The soft-drink giant conducted a survey that suggested consumers would like a new flavor to hit the market. And so it did. And it failed miserably. In 2019 the drink was relaunched in little quantities to promote the Netflix TV series “The Stranger Things”. Although the following example is not about pricing per se, it gets us in the right line of thinking.
The best way to mitigate this discrepancy between what users will do in the real world, and what they think they would do, is to create an environment in which an actual purchase (or other high-commitment action) is involved. In the case of making sure users are willing to invest money, you can often see businesses trying to raise funds through Kickstarter since paying for an unreleased product is a very compelling argument in favor of its viability.
2. You’re testing users outside of their “natural habitat”
The context of the product’s usage matters. The example we like giving most often is that a digital product for ER (Emergency Room) nurses cannot be properly tested in a comfortable office with the help of an unfamiliar to the users device. This means that the further from the truth the context of usability testing is, the less accurate the results are going to be. The solution to this issue is pretty intuitive. Do your best to recreate the context of the app’s usage as accurately as possible.
A similar issue is that users tend to behave differently when they’re being observed. Realizing the purpose of usability testing makes users self-conscious about everything they do. To mitigate this, do your best to emphasize that you’re testing the design, not the user. Additionally, create as much of a relaxed atmosphere as you can. This is often easier said than done, but we usually start with a few minutes of informal small talk. Good soft skills will also go a long way to predispose users to maximum honesty and openness.
3. People generally try to be nice
No one likes to bring bad news. This is certainly true about users when they think they’re critiquing your brainchild, i.e. your design. When giving personal feedback on something users think you are emotionally-invested in, they tend to give positive reviews. Negative feedback is also associated with being rude, which goes against the unwritten norms of social etiquette.
To avoid having your users flatter your product, make sure to distance yourself away from the product as much as possible. Even if it isn’t 100% true, we like to stress the fact that your only task is to test, and that you weren’t the person who created the design. Also, try to highlight the fact that they’re evaluating a design, rather than a person who created it.
4. Users don’t know what they want
Oftentimes, we turn to users when not sure what features to build or whether the feature we’ve built is useful or useable. The instinct to turn to users is right, of course. However, the fruitfulness of the user testing sessions will depend on what questions you ask.
Users are certainly aware of the problems they have. Are they aware of the best solutions? – They might be but don’t rely on it. Asking users something like “what features do you want” is unproductive. Users don’t always know what would be best for them, because it’s our job, as designers to figure this out. This is yet another problem that also stems from poor interview script, but the end result will be unproductive user interview insights.
The remedy to this issue is pretty simple. Ask the right questions. To put it briefly, your questions should be problem-oriented, rather than feature-oriented. This means that instead of asking “what do you need”, ask “what are some of your problems?”. Additionally, try asking open-ended questions that don’t suggest answers. You need your users’ minds as “clean” as possible. Here’s a good article to walk you through which questions are better left unasked.
Outro
Thank you for sticking with us till the end!
This article is anything but a discouragement to talk to your users. It’s an attempt to make you more conscious of the obstacles on the way to insightful and productive user interview/usability testing.
Now that you’re aware of some of the pitfalls of talking to your users, we hope you’ve become product-minded professional!
