Modelling Score Variation in Student Writing with a Big Data System: Benefits, Challenges, and Ways Forward

Abstract

• Aim: This theoretically-oriented research note has multiple aims. First, the note sets out approaches to studying the relationship between linguistic features and essay grades and how this has influenced approaches to score modelling in the domain of student university writing. Second, and more pressingly, the note makes the case for this work to continue by scholars questioning how they have gone about uncovering this language use and how they have used this knowledge to construct models of writing assessment and/or model what lies behind assessment scores. As part of this questioning, the note introduces the method of mixed-effects modelling as a robust alternative to traditional linear regression modelling techniques. This method allows us to determine how the use of language influences assessment scores, while taking account of individual writer and rater variables as well as contextual variables that include task and topic in this modelling process. The note encourages this research in the context of a first-year university writing program in a large U.S. university that has currently set up a “big data” text repository system to allow researchers the opportunity to carry out large-scale corpus examinations of language use. The note concludes by outlining some of the key challenges that scholars need to be aware of when undertaking such work.