Automatically Classifying Students in Need of Support by Detecting Changes in Programming Behaviour

Educational research has established that learning can be defined as an enduring change in behaviour, which results from practice or other forms of experience. In introductory programming courses, proficiency is typically approximated through relatively small but frequent assignments and tests. Scaling these assessments to track significant behavioural change is challenging due to the subtle and complex metrics that must be collected from large student populations. Based on a four-semester study, we present an analysis of learning tool interaction data collected from 514 students and 38,796 solutions to practice programming exercises. We first evaluate the effectiveness of measuring workflow patterns to detect students at-risk of failure within the first three weeks of the semester. Our early predictor analysis accurately detects 81% of the students who struggle throughout the course. However, our early predictor also captures transient struggling, as 43% of the students who ultimately did well in the course were classified as at-risk. In order to better differentiate sustained versus transient struggling, we further propose a trajectory metric which measures changes in programming behaviour. The trajectory metric detects 70% of the students who exhibit sustained struggling, and mis-classifies only 11% of students who go on to succeed in the course. Overall, our results show how detecting changes in programming behaviour can help us differentiate between learning and struggling in CS1.