Many-facet Rasch measurement

Abstract

This chapter provides an introductory overview of many-facet Rasch measurement (MFRM). Broadly speaking, MFRM refers to a class of measurement models that extend the basic Rasch model by incorporating more variables (or facets) than the two that are typically included in a test (i.e., examinees and items), such as raters, scoring criteria, and tasks. Throughout the chapter, a sample of rating data taken from a writing performance assessment is used to illustrate the rationale of the MFRM approach and to describe the general methodological steps typically involved. These steps refer to identifying facets that are likely to be relevant in a particular assessment context, specifying a measurement model that is suited to incorporate each of these facets, and applying the model in order to account for each facet in the best possible way. The chapter focuses on the rater facet and on ways to deal with the perennial problem of rater variability. More specifically, the MFRM analysis of the sample data shows how to measure the severity (or leniency) of raters, to assess the degree of rater consistency, to correct examinee scores for rater severity differences, to examine the functioning of the rating scale, and to detect potential interactions between facets. Relevant statistical indicators are successively introduced as the sample data analysis proceeds. The final section deals with issues concerning the choice of an appropriate rating design to achieve the necessary connectedness in the data, the provision of feedback to raters, and applications of the MFRM approach to standard-setting procedures.