Large Language Models in Cellulose Biopolymer Studies: Evaluating ChatGPT and Microsoft Copilot for Information and Reference Accuracy

Abstract

With the increasing reliance on large language models (LLMs) for scientific research, it is critical to assess their reliability in specialized fields such as biopolymer science, particularly with respect to the verifiability of the references. This study examines the performance of two widely used LLMs, ChatGPT (GPT-4 omni) and Microsoft Copilot (GPT-4), in responding to questions and citing the references for the answers on cellulose biopolymers. The questions are set based on three cognitive levels: beginner, intermediate, and expert, and the accuracy of the responses and references provided by the models are assessed. Results show that ChatGPT outperforms Copilot in all cognitive levels, particularly in addressing mathematical problems. ChatGPT achieves 91.9%, 91.9%, and 88.6% accuracy at the beginner, intermediate, and expert levels, respectively, whereas Copilot achieves 82.4%, 82.5%, and 71.1% accuracy. However, the analysis of references reveals critical shortcomings in both models. While Copilot tends to cite more recent journal articles, ChatGPT often relies on older ones. In many cases, references are incomplete, fabricated, or lack proper context, highlighting the persistent challenge of verifying AI-generated citations. Overall, Copilot outperforms ChatGPT in providing correct references. Results show that ChatGPT provides 6%, 36%, and 45% fabricated references at the beginner, intermediate, and expert levels, respectively, whereas Copilot delivers 13%, 9%, and 7% fabricated references at these levels. This work emphasizes that while LLMs hold promise in supporting scientific inquiry in biopolymers, their current limitations in response accuracy and citation reliability need to be addressed before they can serve as dependable tools for scholarly work.