Comparing Student and Generative Artificial Intelligence Chatbot Responses to Organic Chemistry Writing-to-Learn Assignments

Abstract

Chemistry education research demonstrates the value of open-ended writing tasks, such as writing-to-learn (WTL) assignments, for supporting students’ learning with topics including reasoning about reaction mechanisms. The emergence of generative artificial intelligence (AI) technology, such as chatbots ChatGPT and Bard, raises concerns regarding the value of open-ended writing tasks in the classroom; one concern involves academic integrity and whether students will use these chatbots to produce sufficient responses to open-ended writing tasks. The present study investigates the degree to which generative AI chatbots exhibit mechanistic reasoning in response to organic chemistry WTL assignments. We produced responses from three generative AI chatbots (ChatGPT-3.5, ChatGPT-4, and Bard) to two WTL assignments developed to elicit students’ mechanistic reasoning. Using previously reported machine learning models for analyzing student writing in response to the WTL assignments, we analyzed the chatbot responses for the inclusion of features pertinent to mechanistic reasoning. Herein, we report quantitative analyses of (1) the differences between chatbot responses on the two assignments and (2) the differences between chatbot and authentic student responses. Findings indicate that chatbots respond differently to different WTL assignments. Additionally, the chatbots rarely incorporated the discussion of electron movement, a key feature of mechanistic reasoning. Furthermore, the chatbots, in general, do not engage in mechanistic reasoning at the same level as students. We contextualize the results by considering academic integrity with the assumption that students’ intentions are to engage in academically honest behavior, and we focus on understanding the ethical uses of generative AI for classroom assignments.