Humans vs. large language models: Judgmental forecasting in an era of advanced AI

Abstract

This study investigates the forecasting accuracy of human experts versus large language models (LLMs) in the retail sector, particularly during standard and promotional sales periods. Utilizing a controlled experimental setup with 123 human forecasters and five LLMs—namely, ChatGPT-4, ChatGPT3.5, Bard, Bing, and Llama2—we evaluated forecasting precision through the absolute percentage error. Our analysis centered on the effect of the following factors on forecasters’ performance: the supporting statistical model (baseline and advanced), whether the product was on promotion, and the nature of external impact. The findings indicate that LLMs do not consistently outperform humans in forecasting accuracy and that advanced statistical forecasting models do not uniformly enhance the performance of either human forecasters or LLMs. Both human and LLM forecasters exhibited increased forecasting errors, particularly during promotional periods. Our findings call for careful consideration when integrating LLMs into practical forecasting processes.