Analysis of the challenges in ensuring robust and effective clinical evaluation of AI-based digital medical devices.

Abstract

The rapid evolution of algorithms over time, the validity of the digital criteria used for evaluation and the organisational impact are examples of issues that are both diverse and more specific to digital medical devices (DMDs) than drugs. Furthermore, some DMDs that use artificial intelligence (AI) have the feature of being developed using continuous learning systems, in which the algorithms learn from all the data they use. Despite these unique features, their clinical evaluation often follows existing frameworks designed for drugs. Nevertheless, the specific features of AI-based DMDs call for an adapted clinical evaluation approach and raise new methodological questions. We propose a narrative review of these characteristics and the challenges they could imply for the evaluation of DMDs. Hence, the definition of users and intended use, the choice of datasets for analytical and clinical validation, the specification of outcomes of interest, the use of real-world data studies in addition to randomised clinical trials, and the choice of an appropriate comparator are key elements that can influence the design, analysis and generalisability of the clinical assessment.