Knowledge fusion in deep learning-based medical vision-language models: A review

Abstract

Medical vision-language models based on deep learning can automatically extract image features and fuse them with text information, which has promoted the rapid development of multimodal medical artificial intelligence. However, the complexity of the medical field requires the model to have a deep professional knowledge background. Therefore, knowledge fusion technology provides a new idea for solving medical vision-language tasks. Different from the existing reviews, this paper systematically sorts out the knowledge fusion methods in medical vision-language models from two unique perspectives: the stage characteristics of knowledge fusion and the task-oriented fusion strategy, and provides a new theoretical framework for research in the field. Firstly, this paper introduces the classification of medical knowledge and its applicable scenarios in detail. Subsequently, we systematically discuss the knowledge fusion algorithm based on deep learning and summarize the four different knowledge fusion stages (data construction, pretraining, feature representation and inference) in the medical vision-language model. In addition, this paper comprehensively analyzes the specific strategies of knowledge fusion in five types of medical vision-language tasks (medical report generation, medical visual question answering, medical language-guided segmentation, medical multimodal pretraining, and multimodal large language model), and summarizes the evaluation methods based on knowledge fusion in detail. Finally, we summarize future research directions, including enhanced interpretability, mixture-of-experts models, knowledge editing, etc., aiming to provide researchers with references that have both theoretical value and practical significance.