Analysis of rater effect in the evaluation of second language grammatical knowledge in the context of writing: Application of a generalized linear model

Abstract

Kim, Hyun Jung, Junkyu Lee, and Hyun-Jo You. 2019. Analysis of rater effect in the evaluation of second language grammatical knowledge in the context of writing: Application of a generalized linear model. Linguistic Research 36(Special Edition), 25-57. Despite ample testing research involving the impact of rater background characteristics (e.g. raters’ native language) on assessment, relatively little has been known about how raters’ linguistic knowledge influences their scoring decision-making process. By associating second language acquisition research with regard to two types of linguistic knowledge (i.e. implicit and explicit knowledge), this study aims to explore the influence of raters’ linguistic knowledge and other rater factors (including teaching experience) on second language writing assessment, particularly in relation to the grammatical aspect of performance. Forty-two raters’ grammatical knowledge was measured with an untimed paper-and-pencil grammar test (explicit knowledge) and a timed computer-based experiment (implicit knowledge). The raters also responded to a background questionnaire. For scoring, the raters evaluated five examinees’ responses on two writing tasks (a personal essay and an argumentative essay), based on a six-point analytic scoring rubric. An analysis of a cumulative link mixed model revealed that the raters’ scores could be significantly related to their grammatical knowledge and teaching experience. In addition, different effects emerged across the examinees’ ability levels. These findings contribute to a better understanding of the predictors with respect to rating performance and provide practical implications for rater selection and training in second language performance assessment. (Hankuk University of Foreign Studies · Seoul National University)