Assessing the Effect of Programming Language and Task Type On Eye Movements of Computer Science Students

Abstract

Background and Context: Understanding how a student programmer solves different task types in different programming languages is essential to understanding how we can further improve teaching tools to support students to be industry-ready when they graduate. It also provides insight into students’ thought processes in different task types and languages. Few (if any) studies investigate whether any differences exist between the reading and navigation behavior while completing different types of tasks in different programming languages. Objectives: We investigate whether the use of a certain programming language (C++ vs. Python) and type of task (new feature vs. bug fixing) has an impact on performance and eye movement behavior in students exposed to both languages and task types. Participants: Fourteen students were recruited from a Python course that taught Python as an introductory programming language. Study Method: An eye tracker was used to track how student programmers navigate and view source code in different programming languages for different types of tasks. The students worked in the Geany IDE (used also in their course) while eye tracking data was collected behind the scenes making their working environment realistic compared to prior studies. Each task type had a Python and C++ version, albeit on different problems to avoid learning effects. Standard eye tracking metrics of fixation count and fixation durations were calculated on various areas of the screen and on source code lines. Normalized versions of these metrics were used to compare across languages and tasks. Findings: We found that the participants had significantly longer average fixation duration and total fixation duration adjusted for source code length during bug fixing tasks than the feature addition tasks, indicating bug fixing is harder. Furthermore, participants looked at lines adjacent to the line containing the bug more often before looking at the buggy line itself. Participants who added a new feature correctly made their first edit earlier compared to those who failed to add the feature. Tasks in Python and C++ have similar overall fixation duration and counts when adjusted for character count. The participants spent more time fixating on the console output while doing Python tasks. Overall, task type has a bigger effect on the overall fixation duration and count compared to the programming language. Conclusions: CS educators can better support students in debugging their code if they know what they typically look at while bug fixing. For new feature tasks, training students not to fear edits to learn about the code could also be actively taught and encouraged in the classroom. CS education researchers can benefit by building better IDE plugins and tools based on eye movements that guide novices in recognizing bugs and aid in adding features. These results will lead to updating prior theories on mental models in program comprehension of how developers read and understand source code. They will eventually help in designing better programming languages and better methods of teaching programming based on evidence on how developers use them.