Evaluating and advancing large language models for nanofiltration membrane knowledge tasks

Abstract

Nanofiltration (NF) is a rapidly growing field, resulting in a surge of publications with diverse focuses. It's challenging for researchers to quickly find key information from the vast amount of publications. Large language models (LLMs) have shown promise in analyzing article and reasoning about knowledge in some scientific fields, but their effectiveness in membrane research is unclear. Here, we introduced the first benchmark specifically designed for membrane studies and used it to systematically evaluate six general-purpose LLMs (i.e., Claude-3.5, Deepseek-R1, Gemini-2.0, GPT-4o-mini, Llama-3.2, and Mistral-small-3.1). Our findings revealed that the complexity and depth of NF knowledge pose a significant challenge for these LLMs, leading to poor performance, particularly in tasks involving membrane mechanisms. To enhance LLMs' using in this field, we developed a specialized NF database and integrated it with the LLMs using Retrieval-Augmented Generation (RAG). RAG significantly improved performance across all models, with average gains of 18.5 ​% on Question type tasks and 10.8 ​% on Reasoning type tasks. Moreover, in areas such as membrane fabrication and characterization, several models with RAG demonstrated performance exceeding that of human experts. These results suggested that RAG is a promising strategy for leveraging LLMs in NF research. This study introduced a new path for applying LLMs to membrane research and proposes a professional benchmark to ensure the reliable and effective use of LLMs.