Progress Networks as a Tool for Analysing Student Programming Difficulties

Abstract

The behavior of students during completion of a learning task can give crucial insights into typical misconceptions as well as issues with the task design. However, analysing the detailed trace of every individual student is time-consuming and infeasible for large-scale classes. In this paper, we propose progress networks as an analytical tool to make sense of student data and demonstrate the technique in large-scale online learning environments for computer programming. These networks, which are easily interpreted by teachers, summarise the progression of a student population through a learning task in a single diagram and, importantly, highlight locations where students fail to make progress. Using data from three different programming courses (N > 4000), we provide instructive examples of how to apply progress networks, including how to zoom in on areas of interest to identify reasons for student difficulty. In addition, we propose a simple technique for comparing progress networks across different cohorts of interest, for instance to analyse learning differences between older and younger students, and to investigate learning retention across tasks on the same programming concept. Finally, we discuss options to improve instructional design based on the insights from progress networks, and show that progress networks can also apply to smaller cohorts.