Integrating domain-specific knowledge and fine-tuned general-purpose large language models for question-answering in construction engineering management

Abstract

General-purpose Large Language Models (GLLMs) for Question-Answering (QA) of Construction Engineering Management (CEM) usually lack CEM knowledge and fine-tuning datasets, leading to unsatisfactory performance. Hence, this paper integrates the CEM External Knowledge Base (CEM-EKB) with out-of-domain fine-tuned GLLMs for CEM-QA. It encompasses (i) devising a process to develop the CEM-EKB with 235 documents, (ii) conducting out-of-domain fine-tuning to enhance GLLMs' abilities, (iii) integrating CEM-EKB with fine-tuned GLLMs, (iv) building CEM-QA test datasets with 5050 Multiple-Choice Questions (MCQs) and 100 Case-Based Questions (CBQs), and (v) comparing GLLMs' performance. The results indicate that CEM knowledge-incorporated fine-tuned GLLMs surpass original GLLMs by an average of 27.1 % in professional examinations, with an average improvement of 27.5 % across 7 CEM subdomains and 22.05 % for CBQs. This paper contributes to devising an effective, reusable, and updatable CEM-EKB; revealing the feasibility of out-of-domain datasets for fine-tuning; and sharing a large-scale CEM-QA test dataset.