Can large language models replace human experts? Effectiveness and limitations in building energy retrofit challenges assessment

Abstract

Retrofitting existing buildings is essential to improve energy efficiency and achieve carbon neutrality in the fight against global climate change. Large language models (LLMs) have recently attracted significant attention for their ability to process data efficiently. While LLMs have emerged as useful tools for various tasks, their potential to replace human experts in assessing building energy retrofit challenges remains unexplored. This research explores the potential of replacing human experts with LLMs by evaluating four mainstream LLM chatbots and comparing their performance against a human expert benchmark through semantic similarity and text correlation metrics. It answers the research question: can LLMs replace human experts in assessing the challenges to building energy retrofits? Prompt engineering techniques, including zero-shot and chain-of-thought (CoT) prompting, were employed to guide LLM responses. Results show that LLMs perform well in identifying challenges but are less reliable in ranking them. CoT prompting improves challenge ranking accuracy but does not enhance challenge identification. Incorporating domain-specific knowledge in prompts significantly enhances LLM performance, whereas prompts designed to simulate experts have notable limitations in improving LLM performance. Furthermore, there are no significant performance differences among LLMs, including their advanced versions. While LLMs can streamline the initial identification of building energy retrofit challenges, they cannot fully replace expert judgment in ranking challenges due to their lack of tacit knowledge. This research provides valuable insight into the capabilities and limitations of LLMs in the challenge assessment, offering practical guidance for industry practitioners seeking to integrate LLMs into their building energy efficiency practices.