Context counts: Unveiling the impact of achievement level on teachers’ text assessment

Abstract

Background

Teachers' assessments of student writing proficiency are essential for adaptive instruction and providing feedback. However, studies have revealed a lack of objectivity in text quality ratings of the same text in different contexts, although texts of identical quality should receive identical teacher judgments.

Aims

Our two studies – one experimental and one more ecologically valid – address whether the achievement level impacts teachers’ individual text assessments even when relevant explicit criteria are used.

Samples

In Study 1, participants were 102 student teachers from a university in Germany. In Study 2, participants were 136 student teachers and 91 experienced teachers from Germany.

Methods

In Study 1, participants were randomly assigned to two groups. Student teachers in the high-achievement level group assessed three high-quality texts and one standard text with average quality. Student teachers in the low-achievement level group assessed three low-quality texts and the same standard text with average quality. Participants in Study 2 assessed ten texts randomly drawn from a large corpus.

Results

The comparison of the standard text in Study 1 showed that student teachers in the high-level achievement group assessed it more negatively on a holistic scale and three analytic scales compared to the low-level achievement group. Multi-level data analysis in Study 2 revealed a negative influence of the quality of the achievement level on the text assessment for experienced and for student teachers.

Conclusions

We discuss how the achievement level can bias teachers’ judgments of specific student performances when concrete assessment criteria should be used.