Getting Direct Feedback on Your Product

Shortly after handing out my timer app to some of my friends for testing, one called me and wanted to give feedback. I wasn't prepared. How should I handle this interview? What should I ask? What should I avoid? The short answer is, be as objective and neutral as possible and let the subject speak. The long answer is, that's easier said than done. Let's go into more detail!

Beware of Bias

When you have your own MVP, you want to be the first one to test it. This is usually called dogfooding and helps finding lots of bugs and quirks before the launch. Large companies, like Microsoft, encourage their employees to work with pre-released versions of their software. However, if you and your colleagues, i.e. those who worked on the product, are the only ones who are testing it, you won't get much of a feedback. Your perception is heavily biased by previous knowledge and expectations.

It’s essential having other people, especially your target group, use your product in an early stage. They should not have been exposed to any information about the product or the new feature, e.g. they shouldn't know about your previous design choices, assumptions, metrics, and so forth. Like a group of subjects in a scientific experiment, they should not have been biased.

Thus, when handing over the product to your testers, don't prime them by saying: "I would like you to test our new app which solves problem X by introducing Y". Here you are giving away the clue how your app is intended to work depriving you of the chance of getting new insights whether the subject is using your product as intended and why this might not be the case.

1. Get Rid of Your Filters

Our brain has automatic filter systems which sort out possible irrelevant information so that we focus on the important stuff. This is usually a good thing, e.g. not overseeing a truck when crossing a street. But this implies, in order to be objective, we have to be aware of those filters and consciously avoid filtering details. Psychologist, like me, are trained to consider this so-called experimenter bias or observer bias because in a scientific experiment (not unlike a MVP) they will lead to biased data and wrong conclusions.¹

2. Don't Evaluate. Don't Judge.

The worst thing you can do is immediately judge or raise concerns. "We can't implement it... I already thought about that... This would be too expensive...". This will narrow thinking and reduce the chance of getting new ideas. Be very conscious about the fact that you are not discussing the solutions of a problem. You want to find out more about the problems, goals, and needs the user has.

3. Be as Neutral as Possible

Be mindful about those unintentional and emotional signals you send out, starting from the tone of your voice to the tiny expressions in your face to the gestures you use. If you have an aggressive tone it can be as counter productive as being too excited. First step to control those things is to be aware of them. Make a record of an interview and evaluate your interactions. Then create a short list or post-its of things what you must stop yourself from doing. Have it present when doing your next interview.

4. Don't Set Expectations

According to the Theory of Mind we have a mental representation of others which allows us to be empathic for the needs and feelings of other people.² If you start your interview by saying "Thank you so much for testing. My career depends on the outcome of the product, you know?" your subject will likely spare your feelings and not give you honest feedback. But also subtle details, like the observer's profession, can have a significant influence by creating demand characteristics.¹ For instance, an introduction like "Hi, I am a senior developer working for the tech department" could make your subjects focus on technical details because they assume the interviewer demands that of them.

5. Actively Listen

The main goal of the interview is to gain more insight. Thus, it's crucial to actively listen by letting the interviewee talk, rephrasing his sentences, and asking clarifying questions. Don't think about what you want to say next but what you want to understand. Why didn't they use your main feature as intended? Why didn't they use your product at work? Why did they use a specific feature so often? Behind every action there is usually a goal. For example, I'm not using my timer app to stop the time but to manage my time more efficiently. Try to find out about those hidden motives and needs which allow you to improve your feature to help the user achieve his actual goals.

What the End Result Should Look Like

At the end of your test run, you not only want to have a list of bugs and things to improve but also new ideas about what to do next, based on the insights about the problems, needs, motives, and goals of your potential users. Getting an objective picture by using qualitative data, like an interview, can appear to be cumbersome. However, being aware of your own biases and some training you can gradually improve your skills.