SAMAC-R3-MED: Semantic alignment and multi-agent collaboration of retriever-reranker-responder models for multimodal engineering documents

In the manufacturing industry's lifecycle, a vast amount of engineering documents in text, table, and image formats is generated. Retrieval-augmented generation (RAG) models can enhance retrieval efficiency and adapt to evolving document knowledge. However, challenges in understanding multimodal semantic associations and the absence of engineering-semantic-aligned RAG models result in suboptimal accuracy. This paper introduces a novel approach, namely SAMAC-R³-MED, to tackle these challenges. First, a fine-grained context enhancement strategy is applied to multimodal large language models (MLLMs), bridging multimodal semantic understanding by constructing multi-modal semantic trees (MMST) and multi-modal knowledge graphs (MMKG), forming a hybrid retrieval base. Second, to bridge the semantic gap in RAG models, a new training framework, retriever-reranker-responder (R³), is proposed, utilizing supervised and reinforcement learning with ranking feedback to enhance alignment. Third, a multi-channel hybrid retrieval strategy is implemented for the multi-agent collaboration R³ models, integrating expert feedback, semantic trees, and graphs to optimize the RAG pipeline and improve the accuracy of retrieving multimodal associative semantic contexts. An engineering documents chat (eDoChat) system is implemented, in the case of wind turbine assembly, validating the effectiveness in retrieving and generating accurate multimodal answers. Ablation experiments show R³ models outperform traditional RAG models, and SAMAC-R³-MED achieves state-of-the-art results in multimodal retrieval and generation tasks.