What We Can Learn from Medical Technology Industry About Reliable AI
It is quite rare for my two main areas of consultancy – knowledge management and computerised system validation in the pharmaceutical sector – to overlap, but when it comes to AI…
Whilst many companies are still experimenting with prototypes, the medical sector has already established a strict framework for the safe use of these technologies: Good Machine Learning Practice (GMLP).
These ten principles, developed by leading regulatory bodies such as the FDA (US Food and Drug Administration) and the IMDRF (International Medical Device Regulators Forum), provide a blueprint for robust and trustworthy AI systems, not only in the regulated field of medical technology.
Here are the key GMLP principles:
- Interdisciplinary teams rather than siloed solutions
An AI model is only as good as the understanding of its intended purpose. GMLP requires experts from various disciplines to collaborate throughout the entire lifecycle. Engineers, data specialists and the actual end-users must work together to define the clinical or business benefits the AI is intended to deliver, as well as the risks involved. - Sound software engineering and security
AI is software – and should be treated as such. This means consistently applying best practices in software engineering, cybersecurity and risk management. A methodical design process ensures that decisions are traceable and that the integrity and authenticity of the data are maintained. - Data quality and representativeness
A common mistake in AI projects is a ‘bias’ in the data. GMLP stipulates that datasets must reflect the actual target population. If, for example, only data from a specific user group is used, the model will fail in the real world. Careful data governance is key here. - Strict separation of training and test data
To objectively evaluate a model’s performance, the training and test datasets must be strictly independent of one another. This prevents ‘label leakage’ – a phenomenon whereby the model ‘guesses’ the outcome based on hidden clues in the training data, rather than learning genuine patterns. Only through independent testing can it be demonstrated that the AI also works with new, unknown data. - Suitability of reference standards
Use of the best available (clinical) methods for generating reference data - Tailored model design
The model design must be suited to the available data and the intended use in order to minimise risks such as overfitting - Focus on human-AI interaction
In practice, AI rarely operates entirely independently. GMLP therefore focuses on the interaction between humans and machines. It is essential to ensure that users can interpret the AI’s outputs correctly and that no dangerous over-reliance on the system develops. Transparency and clear information about the system’s limitations are essential for this. - (Clinically) Relevant test conditions
Testing is carried out under conditions that simulate real-world clinical use - Clear user information
Users are provided with transparent information about the system’s capabilities, limitations and updates/re-training. - Control does not stop after roll-out
An ML model is not a static product. GMLP requires continuous monitoring of performance in real-world use (‘post-market monitoring’). This allows performance declines caused by changing data patterns (dataset drift) to be detected at an early stage and risks associated with retraining the model to be managed.
At first glance, applying these principles may seem like a lot of work. However, it is just necessary if we should build trust in AI, isn’t it? And for me the main message is: „It is the controlled data quality, stupid“.
