A Medical Multimodal Large Language Model for Pediatric Pneumonia.

Abstract

Pediatric pneumonia is the leading cause of death among children under five years worldwide, imposing a substantial burden on affected families. Currently, there are three significant hurdles in diagnosing and treating pediatric pneumonia. Firstly, pediatric pneumonia shares similar symptoms with other respiratory diseases, making rapid and accurate differential diagnosis challenging. Secondly, primary hospitals often lack sufficient medical resources and experienced doctors. Lastly, providing personalized diagnostic reports and treatment recommendations is labor-intensive and time-consuming. To tackle these challenges, we proposed a Medical Multimodal Large Language Model for Pediatric Pneumonia (P2Med-MLLM). It was capable of handling diverse clinical tasks-such as generating free-text medical records and radiology reports-within a unified framework. Specifically, P2Med-MLLM was trained on a large-scale dataset, including real clinical information from 163,999 outpatient and 8,684 inpatient cases. It can process both plain text data (e.g., outpatient and inpatient records) and interleaved image-text pairs (e.g., 2D chest X-ray images, 3D chest Computed Tomography images, and corresponding radiology reports). We designed a three-stage training strategy to enable P2Med-MLLM to comprehend medical knowledge and follow instructions for various clinical decision-support tasks. To rigorously evaluate P2Med-MLLM's performance, we conducted automatic scoring by the large language model and manual scoring by the specialist on the test set of 642 samples, meticulously verified by pediatric pulmonology specialists. The results demonstrated the reliability of automated scoring and the superiority of P2Med-MLLM. This work plays a crucial role in assisting doctors with prompt diagnosis and treatment planning, reducing severe symptom mortality rates, and optimizing the allocation of medical resources.