DeepGut: A collaborative multimodal large language model framework for digestive disease assisted diagnosis and treatment.

Abstract

Background: Gastrointestinal diseases have complex etiologies and clinical presentations. An accurate diagnosis requires physicians to integrate diverse information, including medical history, laboratory test results, and imaging findings. Existing artificial intelligence-assisted diagnostic tools are limited to single-modality information, resulting in recommendations that are often incomplete and may be associated with clinical or legal risks.

Aim: To develop and evaluate a collaborative multimodal large language model (LLM) framework for clinical decision-making in digestive diseases.

Methods: In this observational study, DeepGut, a multimodal LLM collaborative diagnostic framework, was developed to integrate four distinct large models into a four-tiered structure. The framework sequentially accomplishes multimodal information extraction, logical "chain" construction, diagnostic and treatment suggestion generation, and risk analysis. The model was evaluated using objective metrics, which assess the reliability and comprehensiveness of model-generated results, and subjective expert opinions, which examine the effectiveness of the framework in assisting physicians.

Results: The diagnostic and treatment recommendations generated by the DeepGut framework achieved exceptional performance, with a diagnostic accuracy of 97.8%, diagnostic completeness of 93.9%, treatment plan accuracy of 95.2%, and treatment plan completeness of 98.0%, significantly surpassing the capabilities of single-modal LLM-based diagnostic tools. Experts evaluating the framework commended the completeness, relevance, and logical coherence of its outputs. However, the collaborative multimodal LLM approach resulted in increased input and output token counts, leading to higher computational costs and extended diagnostic times.

Conclusion: The framework achieves successful integration of multimodal diagnostic data, demonstrating enhanced performance enabled by multimodal LLM collaboration, which opens new horizons for the clinical application of artificial intelligence-assisted technology.