Can medical devices discriminate?

The risk of unconscious bias

Unconscious bias is an issue that can affect the design of any product, including many of the ones we use daily. For example, social media platforms are used by billions of users, however the way they function is often dictated by only a small number of software developers. When the infamous ‘like’ feature was introduced to Facebook in 2009, the developers behind it viewed the feature as simply an innovative way to determine the interests of different users, in order to better tailor the content on their newsfeed. With the designers being predominantly young, male engineers, they did not predict the immense social and mental impacts this small feature would go on to have for the millions of users. It was not long before social media appeared to correlate with depression and anxiety among young people, with many users being negatively impacted in the event they receive especially few ‘likes’ on a post.

The issue of unconscious bias can occur in medical device development as well, without appropriate oversight. For example, designers might introduce their own unconscious biases in the physical design of a device, by not sufficiently testing it on a wide enough variety of users. Such design bias has recently been brought to light in the U.S. for example, with senators calling on the FDA to launch a post-market study of pulse oximeters and skin colour, following reports that these devices have a lower accuracy for patients with darker skin. Though more information is needed to confirm both the accuracy and potential cause of these reports, issues such as these likely point to a lack of appropriate testing or consideration for different user groups during the device development.

Gender bias and artificial intelligence

The introduction of new technologies such as Artificial Intelligence (AI) and machine learning for the diagnosis and treatment of conditions is another cause for concern around design bias. As highlighted by Team Consulting’s Thorbjorg Petursdottir in her article ‘Is AI Sexist?’, there have already been a number of instances where software developers have inadvertently coded bias against certain genders into algorithms.

Such bias could have serious consequences in the medical space, especially when AI is used to help diagnose illnesses or symptoms. A poor data set, such as one based on historical data that favours one gender, could lead to an algorithm diagnosing the favoured gender more accurately, or failing to diagnose the other entirely. If we wish to continue exploring the use of AI and machine learning in healthcare, it will be essential to account for such errors to ensure patient safety.

Design considerations for different user groups

While more research is needed in this area, studies have also found some potential physical differences between different ethnic groups which may need to be accounted for in the development of medical devices. For example, a study conducted on four different ethnic groups used computational fluid dynamics to determine differences in nasal spray deposition. Though conducted on a small number of subjects, the study found the Caucasian and Latin American subjects had a lower patent nasal cavity and lower consistency in particle deposition patterns compared to African Americans and Asians. Differences such as these could well need to be accounted for in the development of drug delivery devices such as nasal sprays, to ensure patients of different ethnicities get the appropriate dosage and treatment they need.

The way forward

Without the right checks and balances in place, unconscious bias can become a significant issue in the design of medical devices. Medical device manufacturers must work to avoid this by drawing on inputs from a wide variety of stakeholder groups throughout a device development, from the engineers creating the device to those testing it. A diligent design team should incorporate human factors approaches throughout the device development as well, employing user studies, design research and analysis to gain insights into how different users interact with a product, as well as their understanding and expectations of it. User studies are essential to identify the strengths and weaknesses of a device, as well as an important way of ensuring that different users are placed at the heart of the design process and preventing discrimination.