Eliciting Course Feedback through a Bug Bounty Program

Abstract

In this paper, we present a bug bounty program that can aid instructors in systematically gathering formative feedback for iterative refinement of their course content. We describe the logistics of implementing the program, explain the types of content in which bugs are reported, elaborate on how students received the program, and evaluate if the program can be effective in improving the course quality. We present data from a large undergraduate Data Structures and Algorithms (DSA) course that was offered consecutively for four semesters. In total, 898 students enrolled in our course, 200 students reported at least one bug, and 373 bugs were reported in total related to incorrect or ambiguous content in instructional material, logistical errors such as broken links, and bugs in short programming problems such as less exhaustive testing. We found that a majority of the students who participated reported a single bug. We also found that the normalized number of bugs reported per student gradually decreased across semesters to almost one-half after two iterations (0.53 bugs reported/student in the first two semesters vs 0.28 bugs reported/student in the last two). This suggests that the program can be effectively used to iteratively refine the course content and improve the learner experience. Students received the program enthusiastically with 97% showing positive or neutral valence on the continuation of the program in future course offerings.