Multi-modal learning methods in medical imaging area: A survey

Abstract

Multi-modal learning is an important branch in the field of deep learning area, which has been widely used for processing data from different media. The fusion of different modalities in natural images has shown significant results, but less attention has been paid to medical images of individual modalities due to data scarcity. The discussion of applications of multi-modal learning has raised great interest in the medical field, including general fusion methods, deep learning-based methods, and large language model-based methods. With the aim of describing the evolution of different models in the field of multi-modal medical imaging, this survey provides a thorough overview of representative methods and related applications. In this study, we first introduced the concept of modality and the development of multi-modal learning, then listed the commonly used medical modalities and fusion strategies. After that, we described the branches of multi-modal models in the medical imaging field in detail, along with various application scenarios and open datasets. We hope our survey will provide guidance for readers to understand typical models and the growing trend within the medical imaging domain.