Effectiveness of retrieval augmented generation-based large language models for generating construction safety information

Abstract

While Generative Pre-Trained Transformers (GPT)-based models offer high potential for context-specific information generation, inaccurate numerical responses, a lack of detailed information, and hallucination problems remain as the main challenges for their use in assisting safety engineering and management tasks. To address the challenges, this paper systematically evaluates the effectiveness of the Retrieval-Augmented Generation-based GPT (RAG-GPT) model for generating detailed and specific construction safety information. The RAG-GPT model was compared with four other GPT models, evaluating the models' responses from three different groups––2 researchers, 10 construction safety experts, and 30 construction workers. Quantitative analysis demonstrated that the RAG-GPT model showed superior performance compared to the other models. Experts rated the RAG-GPT model as providing more contextually relevant answers, with high marks for accuracy and essential information inclusion. The findings indicate that the RAG strategy, which uses vector data to enhance information retrieval, significantly improves the accuracy of construction safety information.