How do teachers process technology-based formative assessment results in their daily practice? Results from process mining of think-aloud data

Abstract

Background

Technology-based formative assessments are considered promising in terms of reducing teachers’ workload and providing validity advantages but little is known how teachers use the assessment results to inform their instruction in their daily practice.

Aims

We explored how teachers process technology-based formative assessment results using think-aloud methodology in an ecologically valid setting.

Sample

Forty-eight experienced in-service teachers participated in the study.

Methods

We asked the teachers to verbalize their thoughts while they processed students’ formative assessment results as they usually do. Screencasts of the verbalizations and assessment results were recorded. Based on these, trained raters coded the main steps of processing and which specific aspects of the results were noticed based on a deductive-inductive coding scheme. Cluster analyses were applied to explore differences among teachers, and process mining was conducted to explore the main processes.

Results

We found four main steps: noticing results, comparing with personal perspective, analyzing errors and constructing instructional implications. Relative durations of these steps vary substantially among teachers. Cluster analyses indicate that processes were differentiated according to the complexity of summarizing and building relationships between single data points. The fitted process model revealed low dependency values in general and indicates that noticing results on its own seemed to be insufficient for constructing instructional implications.

Conclusions

This study generates the hypothesis that analyzing errors and comparing results with the personal perspective are important for teachers for next instructional decisions.