Multimodal radiology AI

Abstract

The growing armamentarium of artificial intelligence (AI) tools cleared by the United States Food and Drug Administration mostly target a narrow, single imaging modality or data source of information. While imaging technologies continue evolving rapidly, it is recognized that multimodal data provides synergistic information and enables better performance than what is achievable when these modalities are used separately. Deep learning approaches can integrate multimodal data, including not only imaging but also non-imaging modalities such as electronic medical records (EMRs) and genetic profiles. Such convergence advances clinical applications and research for improved effectiveness, especially the prediction of disease risks. This new avenue could address concerns over justification of imaging scans, clinical context-based interpretation of examinations, effectiveness of single modal and multimodal data to influence clinical decision making, as well as prediction of personalized disease risk. In this new era of radiology AI, the paradigm is being shifted from imaging alone AI analytics to multimodal artificial general intelligence (AGI). The heterogeneity of the data and the non-intuitive nature of certain modalities pose major challenges for developing multimodal large AI models and at the same time bring enormous opportunities.