Differential Assessment, Differential Benefit: Four-year Problem Roulette Analysis of STEM Practice Study

Abstract

Using five million responses to thousands of practice examination questions on an optional study service known as Problem Roulette, we explore subject-specific differences in assessment style, grade benefit from usage of the service, and differential features in study behavior and grade outcome by birth sex. Our study includes more than 20,000 students enrolled in eight terms of introductory courses in general chemistry, physics and statistics. Student responses in the space of accuracy and response time reveal domain differences; by these measures, physics problems are typically both more difficult and more complex. Grouping students by term-length practice volume, we find significant positive grade benefits to higher volumes of study in chemistry and statistics. Across all subjects, we find that females gain more grade benefit from high study volume than males. Female students also outwork males during prime study hours yet, on average, earn 0.13 ± 0.03 lower grade points in chemistry than males with the same response accuracy in practice, with null results in statistics and physics.