Integration of Multi-Source Medical Data for Medical Diagnosis Question Answering

Medical question answering aims to enhance diagnostic support, improve patient education, and assist in clinical decision-making by automatically answering medical-related queries, which is an important foundation for realizing intelligent healthcare. Existing methods predominantly focus on extracting key information from a single data source, e.g., CT image, for answering. However, these methods are not enough to promote the development of intelligent healthcare, because they lack comprehensive medical diagnosis capabilities, which usually require the integration of multi-source data (e.g., laboratory tests, radiology images, pathology images, etc.) for processing. To address these limitations, our paper introduces the extended task of medical question answering, named medical diagnosis question answering MedDQA. MedDQA task aims to answer questions related to medical diagnosis based on multi-source data. Specifically, we introduce a corresponding dataset that incorporates multi-source diagnostic information from 250,917 patients in clinical data from hospital records, and utilize a large-scale model for constructing Q&A pairs. We propose a novel system based on large language models, named medical multi-agent (MMA) system, which includes a mechanism of multiple agents to handle different medical tasks. Each agent is specifically tailored to process various modalities of data and provide outputs in a uniform textual modality. Experimental results demonstrate that the MMA system’s architecture significantly enhances the handling of multi-source data, thereby improving medical diagnosis, establishing a robust baseline for future research.