Harnessing code domain insights: Enhancing programming Knowledge Tracing with Large Language Models

Abstract

Knowledge Tracing (KT) evaluates students’ mastery of knowledge by analyzing their historical interactions with exercises and predicts their performance on subsequent tasks. Although traditional KT methods have begun to focus on the assessment of programming skills, they are limited by the bottleneck of manually annotating knowledge concepts (KCs) and the inadequacy of constructing relationships between these points. To address this issue, we propose a Knowledge Tracing method Enhanced by the powerful Code insight capabilities of Large Language Models (CEKT). Specifically, we designed three different prompt tuning strategies for Large Language Models (LLMs) to comprehensively construct Q-matrices that cover KCs and their relationships across various programming domains and exercises. Additionally, we developed a knowledge graph integrating three dimensions to express the complex relationships between KCs in a fine-grained manner, thereby providing a more accurate assessment of students’ knowledge mastery. Furthermore, we established a graph attention network among KCs to promote interaction between representations of similar syntactic KCs, enhancing the inference capability of students’ programming knowledge state and the effectiveness of KT. Through this approach, we achieved high-quality and interpretable knowledge state inference and demonstrated outstanding performance in predicting student outcomes. Our work highlights a potential future research direction for prompt-tuned LLMs in the KT domain, emphasizing high interpretability and efficiency. For broader research purposes, we have prepared to release our data and source code at https://github.com/xinjiesun-ustc/CEKT, encouraging further innovation in this field.