Don’t listen to users (and 3 other human factors tips)

- Written by Aidan Hyde & Eric Sugalski

Defining the right use case, ensuring a positive user experience, and mitigating user related risks are critical parts of the medical device design process. Interfacing with users (often) is a smart thing to do, especially when developing a new-to-the-world device.

However, user feedback may lead you astray if you aren’t collecting and processing data the right way. Here are four human factors tips to ensure you get the most out of your research.

Tip #1: Don’t listen to users

Say what? Doesn’t this go against good research practice?

People are poor sources of information when reflecting on their past experiences. Most of us can’t remember what we had for dinner two nights ago. Asking research participants to recall events from two months ago is very likely to generate unreliable data.

Additionally, when being interviewed, participants want to be seen as experts. So, they are often reluctant to say, “I don’t know” -- even when they don’t. As a result, some of the information you hear will be questionable at best.

Along the same lines, participants are often reluctant to admit when they have problems with products. It may make they them seem less competent than peers that can operate those same products just fine. Participants want to maintain the expert persona, which may prevent you from hearing about the challenges that actually exist.

So, you read it right –don’t rely too much on your ears. BUT, make sure your eyes are wide open. More on that next.

Tip #2: Use mockups, not prototypes

You have a new idea for a medical device. And you want to get some feedback on it. Most companies in this situation will build a prototype and start conducting demonstrations.

Big mistake.

First, when you are exploring use cases, ergonomics, interfaces and other product usability aspects, you want people to be focused on their experience with the concept and how it affects them. If you serve up a fully functional prototype, they will be mesmerized (or more likely, confused) with the tech. It will be difficult for you to redirect the conversation toward the user-product interface and outcomes associated with it.

So, steer clear of functional prototypes when conducting user research at the early stage.

Keep your mockups loose and rough. Use foam core, 3D prints, and found objects. Think of concept mockups as “props” – ones that allow users to “act-out” scenarios.

Ideally, have anatomical models or other simulators on-hand. These supplements will allow the “act-out” sessions to be more realistic and will provide greater context on the procedural challenges and constraints.

Ask the user to walk through how they would envision the mockup being used as an alternative to the standard of care. Of course, pay attention to the primary medical task, but also have the user walk through secondary tasks. These can include storage or transportation, powering-up and charging, calibration, cleaning, etc. Often, these secondary tasks are where most time is spent and inconvenience is incurred.

Don’t go into a user interview with a single mockup. Go in with several. Here’s why:

If you have three mockups and ask users to “act out” each of them, you’ll be able to spot differences in ergonomics, intuition, efficiency, and more. Additionally, you’ll be able to ask users about their most and least preferred mockups -- and WHY. Three mockups will allow for comparisons, which always leads to new insights.

By contrast, if you have a single mockup, the information you’ll receive will be much more limited. Most participants will be reluctant to share criticisms because it’s your idea – your baby. They don’t want to insult you or your baby. So, they’ll be reserved with their comments, and they’ll most likely tell you want they think you want to hear.

Tip #3: Keep your interviews structured, but malleable

When conducting user studies, think of it as data collection. You do need to have some scientific rigor to capture reliable data. So, write a protocol. Standardize questions. Randomize the order in which you present mockups. Avoid introducing biases.

About half the time you spend interviewing should be asking consistent questions from your protocol – capturing data that you can then compile and synthesize later.

The other half of the time should be reserved for wandering. Reason being -- you probably don’t know the right questions to ask. So, give some time and space for interviews to evolve. When a question sends a conversation on a tangent, see where it goes. Maybe it illuminates a workaround for a clunky feature. Maybe it steers toward a product for a different procedure that has some interesting similarities.

Space to wander will open up topics that can be delved into more deeply during subsequent user sessions.

Tip #4: Seek invalidation, not validation

User research needs to be about “learning” – not “proving”.

Unfortunately, many companies treat user research as the opportunity to validate the direction that they have already pre-selected. In cases like these, research is the “final check” on their concept – one that perhaps is needed to satisfy the board or the investors. This approach is always a missed opportunity.

Instead, user research needs to be focused on uncovering the problems, the challenges, the criticisms. Teams should be seeking invalidation, not validation. The invalidating data is much more informative to the design process than head nods of affirmation. The invalidating data will clue you into the negative perceptions, concerns, fears -- the emotions triggered by your current concept (or the next one).

To this end, it can be valuable for companies to leverage third parties when conducting user research. It injects a level of objectivity and concept neutrality into the process that is difficult for concept originators to maintain.

If you are seeking support in exploring concepts, assessing usability, and conducting human factors studies, Archimedic may be able to help. Reach out if you would like to explore a collaboration.