A Facets Analysis of Analytic vs. Holistic Scoring of Identical Short Constructed-Response Items: Different Outcomes and Their Implications for Scoring Rubric Development.

Abstract

In scoring short constructed-response items it may be possible to apply different rubric types depending on the trait of achievement assessed. A rating scale and a partial credit Many-Facet Rasch Models (MFRM) were used to investigate whether levels-based (holistic) and hybrid (analytic) scoring rubrics functioned interchangeably when scoring short-response English reading comprehension test items. Whereas most research in similar contexts has focused solely on rater reliability, the use of MFRM in this study enabled examination of both the reliability and rating scale functioning aspects of scoring rubrics in parallel. It also enabled consideration of their effects on item and examinee parameters. This more comprehensive approach allowed the findings to be linked to general notions of rubric construct-relevance and score interpretation, and to demonstrate an approach to generating evidence for a more balanced consideration of advantages and disadvantages of each rubric in terms of both reliability and validity.