International partnership for governing generative artificial intelligence models in medicine

Abstract

Generative artificial intelligence (GenAI) models, such as generative adversarial networks (GANs) and transformer-based large language models (LLMs), are developing at an accelerated pace and positioned to be integrated into clinical workflows and healthcare systems across the world. However, this rapid rise of GenAI in medicine and healthcare presents not just unprecedented opportunities, but also systemic risks in the integration of this new technology and critical vulnerabilities in terms of safety, governance and regulatory oversight. GenAI and LLMs are non-deterministic in nature, possess broad generalist functionalities, and display evolving capabilities¹. These characteristics challenge conventional regulatory frameworks designed for deterministic, task-specific artificial intelligence (AI) models, such as those for Software as a Medical Device (SaMD).

Some of the fundamental risks associated with GenAI and LLMs applications in healthcare are clear but yet to be fully addressed by current regulatory framework (‘known unknowns’), whereas other risks and challenges have not yet even surfaced (‘unknown unknowns’). Known unknowns include a lack of transparency in training data (including the possible use of synthetic data for training²), susceptibility to bias, hallucination of incorrect medical content, and potential misuse in high-stakes clinical settings¹ (Box 1).