What's the Point? How Scores Undermine Written Comments on Open-Ended Work

Scaling assessments typically relies on quantifying work quality, yet written comments are the assessment method of choice for open-ended work. Existing scalable solutions compromise by mapping quality scores to pre-authored comments, but how scores influence interpretation of these comments is not well understood. We report results from a study of how 441 participants authored and revised short stories in response to a score, written comments, both types of feedback, or no feedback. We analyzed data from the story-writing task and two surveys to determine task and feedback satisfaction, revision depth and effort, and improvement between drafts for each participant. We found task satisfaction and task performance were positively correlated among participants who were shown a score. Feedback satisfaction, revision effort, and improvement were highest among participants shown written comments. Either type of feedback prompted more deep revisions than no feedback, but together elicited fewer deep revisions than written comments alone. Our work informs the design of scalable open-ended assessment systems by contributing insights regarding how scores influence perceptions of written feedback and subsequent revision outcomes.