Developing machine learning and AI for medical devices

A quick history of AI in healthcare

The term artificial intelligence was first coined at the Dartmouth Summer Project in 1956 where researchers from different fields gathered and produced ideas, papers and concepts on their views of AI at that time. Funding for AI research was sporadic, from highs in the late 1950s and early 1960s to lows during the 1970s.

As early as the 1970s, researchers saw the benefits of AI in healthcare. During this era four different AI systems emerged from Pittsburgh, Rutgers, Stamford and MIT:

• INTERNIST 1 was a system for differential diagnosis in internal medicine.
• MYCIN was a system for infectious disease therapy assistance. During an independent assessment it was found to score highly on both accuracy of diagnosis and effectiveness of treatment.
• CASNET was used in consultation of glaucoma.
• PIP was an early diagnostic tool used in the evaluation of patients with oedema.

Already at that time, the potential of AI in healthcare had been realised and specifically how it could benefit diagnostics. Fast forward 50 years, and we have AI technology that can interpret a medical image up to 10,000 times faster than the average radiologist, machine learning models that can assess the severity of Covid-19 from chest X-rays, and AI that can improve patient diagnosis and care by augmenting digital pathology. There is still a long way to go before this technology is fully optimised and able to handle the full complexities of human health. For example, IBM’s Watson Health, launched in 2015 to diagnose patients and recommend treatment options, was reported by customers to spit out erroneous cancer treatments. With the huge impact that AI in medical devices can have on patient lives, it is important that these technologies are developed and regulated with great care to ensure patient safety.

Key considerations in the design of machine learning in medical applications

Design considerations

With the recent hype around AI, it is important to consider when AI should be applied to a problem. This is especially the case in the safety critical and highly regulated world of healthcare, where the technology must be accepted by multiple stakeholders, from clinicians to patients, regulators and payers. Any new AI tool will need to align to all of these stakeholder requirements. For example, it may need to be able to integrate well into a clinical setting, be easy to use for patients, and have safety critical algorithms that are transparent and explainable to regulators.

Some useful checks are provided below to help inform the AI design process:

1. 1. Does it solve a real problem – something that all stakeholders (clinicians, patients, regulators and payers) identify with?
   2. Does it use the principle of ‘Design Thinking’ to understand feasibility, viability and desirability?
   3. Does it use many of the research techniques of front end innovation to confirm that AI is the right solution for a problem, rather than a technique without a specific purpose?

Gathering and accessing data for the machine learning model

A key consideration for the development process is the strategy for gathering and curating data. If the dataset is bad quality (e.g. not inclusive or too small) it will affect the performance of the ML model significantly. In the healthcare setting, data can be sensitive, difficult to obtain and potentially distributed across multiple different digital systems.

Evaluating the machine learning model

It is important to not train the model on the whole dataset to prevent overfitting and leave a subset of the dataset for testing how well the model is performing. A common train/test split is typically to use 70% of the data for training and 30% for testing or even to split the data into train/test/validation and use 60% for training, 20% for testing and 20% for validation. The validation dataset is used to evaluate the model as it is being built to find the best parameters. The testing data is then used to test how well the model performs on data it hasn’t seen before.

When evaluating the model to check for bias a good test would be to take a subset of the test data (only including women for example) and check whether there is a higher percentage of false positives and false negatives than when looking at the entire test data.

Regression metrics are quite different from classification metrics as regression models are predicting a continuous range instead of a discrete class. Common metrics for regression models are explained variance, mean squared error and R².

It is also important to evaluate the model in a clinical setting different from the setting in which the data to train the model was gathered. A research group at the Icahn School of Medicine at Mount Sinai had developed an algorithm to identify pneumonia in lung X-rays. The model performed with greater than 90% accuracy on X-rays at Mount Sinai, but when they tested their model at different institutes it was far less accurate. They later found out that the model had developed a relationship to the odds of how common pneumonia was at each institution, which was something the model should not have done. It can be easy to miss little factors that the model is considering when it is being trained on data gathered at one location, using one group, at one time. This leads us back to the importance of understanding the data, problem and the solution being implemented, which brings us on to explainability in AI-based medical devices.

AI in medical devices: explainability

As well as being accurate, the model needs to be explainable, as clinicians, regulators and most likely patients will want to understand the basis on which AI-based medical devices are making their decision.

An example of the need for explainability was demonstrated when Rich Caruana, with other researchers, was working on a project to better understand pneumonia. The project involved using a machine learning system to decide whether patients should be kept in the hospital overnight or sent home after they are first diagnosed. One of the models used on the data had found a relationship between having asthma and being low risk, resulting in the patient being treated as an outpatient. The model arrived at this relationship because asthma patients were considered such a high risk that they were immediately put into the ICU and critical care, resulting in them being less likely to die from pneumonia. This potentially life-threatening rule that the machine learning algorithm had developed could be hard to find or correct. Likewise, it highlighted that there could be other harmful relationships that the machine learning algorithm had found going undetected.

Regulators do already require explainability and the GDPR requires that if AI makes a sufficiently important decision about something then an explanation needs to be provided when that decision is made. Those working in AI medical device regulation should therefore work with other regulatory bodies, such as legal, before releasing AI tools to the market and developers should work closely with clinicians to integrate new AI tools into a clinical setting, to make it more understandable for everyone.

For further information on good machine learning practices (GMLP) for developing ML and AI-based medical devices, the FDA and MHRA have identified 10 guiding principles that can be found on the UK Government website.

AI medical device regulations

The benefits of AI in healthcare have already been realised by the research community, with over 12,000 life science papers describing AI/ML already published. However, making the most of the benefits of ML and AI-based medical devices seems to be lagging, largely due to the strict regulations that apply to medical devices.

The FDA has put together a list of AI/ML enabled medical devices that are marketed in the United States. At the time of writing, there are currently 343 medical devices with AI/ML capabilities on this list, with 93.5% having arrived on the market between 2015 and 2021.

Around 70% of those AI-based medical devices are used for diagnostic imaging, indicating that AI in diagnostics is a real trend. The FDA states that the AI/ML devices that have received approval involved “locked” algorithms, meaning that given the same input, the model will always give the same output. However, with this surge of AI/ML enabled medical devices hitting the market, there still seems to be reluctance towards adopting these solutions, possibly due to regulatory framework hindrance or lack of trust in these technologies by patients or practitioners.

Current ML/AI medical device regulations

Currently, there are no harmonised standards that regulate the use of machine learning in medical applications and devices and companies are not required to classify their technology as AI/ML based. Despite this, many AI-based medical devices that use “locked” algorithms, (algorithms that will not change beyond the point of regulatory submission), have been approved for use in the EU and USA by adhering to the medical software regulations already in place.

Regulations for AI

What to keep in mind when regulating an AI/ML enabled medical device:

• Data gathering:
  How was it determined that enough data was used to train the model?
  How was bias eliminated?
  How was invalid or missing data handled?
  How was confidentiality and integrity of the data, as is required by the GDPR, handled?
• Development of software for gathering data, labelling, training and testing must be validated under computerized system validation (CSV) according to ISO 13485:2016 4.16.
• Machine Learning libraries to pre-process the data and train the model are usually not part of the medical device, so IEC 62304 is not applicable if that’s the case. However, any machine learning libraries or Software of an Unknown Provenance (SOUP)s / Off The Shelf Software (OTSS)s used to train the model need to be validated and proved to fit the best parameters for the selected model.
• The trained model is considered part of the medical device and is responsible for making the correct prediction. The model needs to be verified to demonstrate that the trained model and the machine learning library used (for example a model with its parameters and predicted function), give the correct output mathematically. The validation of these machine learning libraries then, for example, include defining the requirements of the functions of the machine learning libraries used and verifying that the corresponding functions meet those requirements.
• The validation of the medical device includes determining the accuracy of the device to predict outcomes, i.e., correctly detecting cancer in an image.

Proposed action plan

The FDA has proposed a regulatory framework for the modification to AI/ML based software as a medical device as well as an action plan for AI/ML software as a medical device, in 2019 and 2020 respectively. As current regulation frameworks do not account for adaptive or continuously learning algorithms, this proposed plan intended to describe a total product lifecycle-based regulatory framework. This would allow for modification to be made to the algorithm while it is learning in the field, without affecting the safety or the effectiveness of the ML or AI-based medical device.

Such adaptive AI could have great benefits in healthcare, if regulated effectively to ensure safety. An adaptive AI has the ability to continually train itself on real data, instead of being limited to historical data. Such learning could help improve diagnosis of rarer diseases for example, as more patients are analysed in real time. With a clearer pathway for FDA approval outlined, it is only a matter of time before the first medical device with adaptive AI software is FDA and/or EU approved.

The future of AI in diagnostics

By improving current regulations for AI/ML based software as a medical device, as well as providing a regulation framework for those technologies with continuously learning algorithms, we can facilitate the development and adoptions of these increasingly popular technologies which will have the power to learn from real-world experience and improve healthcare. Having a clear regulatory framework and good development practices of AI in medical devices will help to ensure transparency of these algorithms and gain the trust of patients and practitioners to adopt them.