Peer instruction: do students really learn from peer discussion in computing?

Peer Instruction (PI) is an instructional approach that engages students in constructing their own understanding of concepts. Students individually respond to a question, discuss with peers, and respond to the same question again. In general, the peer discussion portion of PI leads to an increase in the number of students answering a question correctly. But are these students really learning, or are they just "copying" the right answer from someone in their group? In an article in the journal Science, Smith et al. affirm that genetics students individually learn from discussion: having discussed a first question with their peers, students are better able to correctly, individually answer a second, conceptually-related question. We replicate their study, finding that students in upper-division computing courses (architecture and theory of computation) also learn from peer discussions, and explore differences between our results and those of Smith et al. Our work reveals that using raw percentage gains between paired questions may not fully illuminate the value of peer discussion. We define a new metric, Weighted Learning Gain, which better reflects the learning value of discussion. By applying this metric to both genetics and computing courses, we consistently find that 85-89% of "potential learners" benefit from peer discussion.