Multi-head divide-and-conquer residual-attention mechanism with pointer network for multimodal question summarization in healthcare

Abstract

In contemporary medicine, summaries of medical questions are vital for effective and precise patient care. Current techniques handle only text-based summarization without considering the merit of incorporating visual information. To meet this, this research presents a multimodal summarization system that combines textual queries with medical images to support the extraction of meaningful details. The proposed system has three phases. In the first step, a gradual fusion decoder bidirectional encoder representation from transformers with vision transformers is utilized to produce fine-grained feature maps and diagnose diseases. The Multi-Agent Contextualized Diffusion Model (MACDM) is then utilized to contextualize knowledge using cross-modal information. Lastly, a Multi-head Divide-and-Conquer Residual-Attention mechanism with Pointer Network (MDCRAPN) is utilized to provide brief and relevant summaries. Furthermore, the hermit crab shell exchange algorithm is integrated to optimize hyperparameters for improved performance. The experimental results indicate that this proposed approach performs better than existing approaches with a recall-oriented understudy for gisting evaluation-1 score of 48.11 on the Multimodal Medical Question Summarization (MMQS) dataset. This approach significantly enhances the identification and summarization of medical disorders, demonstrating the potential to enhance healthcare communication and decision-making.