Treating engineering students equally without treating them equally: exploring the multi-evaluator paradox of a capstone engineering design project

In engineering curricula, a final capstone project connects to daily practice in an (as much as possible) authentic setting, to test and prepare students for careers in the capital-intensive chemical engineering field. This paper examines a particular aspect of using capstone engineering design projects for experiential teaching: examination and evaluation of students’ practical work in firms. Such projects are as diverse as they are complex; therefore, performance evaluations of these projects tend to be particularly complicated. These projects do not typically correspond well to a singular set of performance criteria or rubrics, and therefore, a degree of inter-evaluator bias is to be expected. Drawing on a content analysis of the qualitative evaluations of individual design projects, completed by several cohorts of graduates of an engineering program, we uncover a phenomenon, which we call the multi-evaluator paradox of design projects. This phenomenon holds that statistically speaking, all examiners of these projects grade these projects essentially similarly, i.e. there is no significant inter-grader bias of both final and sub-category grades. Simultaneously, however, the verbal motivations of these grades vary greatly between examiners, even when there are structured rubrics underpinning their assessments. Using Computational Text Analysis (CTA), we examine this thus-far undocumented phenomenon and offer possible explanations for its existence. We question whether the existence of this phenomenon is detrimental to educational quality, and we provide suggestions for managing these consequences.