MERMaid: Universal multimodal mining of chemical reactions from PDFs using vision-language models

Abstract

Data digitization of scientific literature is essential for creating machine-actionable knowledge bases to advance data-driven research and integrate with self-driving laboratories. It is especially critical to extract, interpret, and structure data from graphical elements, the primary medium for conveying complex scientific insights. However, this remains challenging due to the inherent lack of semantic structure in the prevalent PDF format, the complexity of visual content, and the need for multimodal integration. We present MERMaid (multimodal aid for reaction mining), an end-to-end pipeline that converts disparate visual data across PDFs into a coherent knowledge graph. Leveraging the emergent visual cognition and reasoning capabilities of vision-language models, MERMaid demonstrates chemical context awareness, self-directed context completion, and robust coreference resolution to achieve 87% end-to-end accuracy across three chemical domains. Its modular design facilitates future application to diverse data beyond reaction mining, promising to unlock the full potential of scientific literature for knowledge-intensive applications.