Exercise-Aware higher-order Thinking skills Assessment via fine-tuned large language model

Higher-order thinking Skills (HOTS) are complex cognitive skills that go beyond the basic levels of memory and comprehension. It is critical to developing an individual’s critical thinking and problem-solving skills. Current methods for assessing HOTS typically rely on expert judgment or specially designed assessment tasks, which have become well-established and reliable paradigms but are time-consuming and difficult to transfer. Accordingly, this study aims to develop an automated HOTS assessment model that will reduce reliance on experts and enhance efficiency and accuracy. Large language models (LLM) are pre-trained models using deep learning techniques to deal with natural language processing tasks. They have excellent knowledge base and reasoning capabilities, and researchers have applied them to a range of domains. In this paper, we proposed the Exercise-Aware higher-order Thinking skills Assessment (EATA) model based on fine-tuning the LLM. The EATA comprises the Exercise Awareness (EA) and the HOTS Assessment (HA) modules. The EA module includes a pre-trained Text2Vec and a multilayer perceptron (MLP). It integrates the exercise text information with the HOTS labeling information to generate the higher-order exercise embedding matrix. The HA module employs a pre-trained LLM as the underlying network, which takes the student’s learning records with the higher-order exercise embedding as inputs and automatically assesses the student’s HOTS through a fine-tuning technique. In this way, EATA can emulate traditional assessment methods, but replace experts with LLM. It reduces the interference of human factors, thus improving efficiency. To verify the validity of EATA, we collect 43070 online exercise data from 181 undergraduate students in a university. The experiments show that EATA can effectively assess students’ HOTS, indicating the potential value of LLM in HOTS assessment tasks. The implementations are available at https://github.com/xxiongGG/EATA.