Evaluating Educational Intervention Fidelity with TranscriptSim, A Replicable NLP Technique

Abstract

Pedagogical implementation research at large identifies and addresses factors affecting adoption and sustainability of evidence-based practices. Part of this work is elaborating and testing implementation strategies which prescribe educator intervention techniques and processes to adopt and integrate into educational settings. However, in practice, implementation research is slowed and constrained by the temporal and monetary costs of conducting manual evaluations. In this study, we use an automated, low-cost, and scalable natural language processing (NLP) approach we call TranscriptSim to assess intervention fidelity in a teacher coaching study (TeachSIM). TranscriptSim quantifies similarity between the intervention protocol and intervention transcripts as an approximation of coaches’ fidelity to the intervention protocol.