A novel application of multilevel SEM: Teaching quality as mediator between intervention and student achievement

Abstract

: Improving teaching quality has been an ongoing pursuit for policymakers, researchers and broader government agencies and organisations. Professional development (PD) initiatives directed at teachers are a recognised strategy for improving teaching quality, with the underlying intention to support improved student achievement. Yet there is limited evidence of the actual impact of PD directed at teachers on student learning outcomes. Quality Teaching Rounds (QTR), a well-recognised form of collaborative PD in Australia, is one of few approaches to provide evidence of impact on teaching quality and student outcomes. Through multiple Randomized Controlled Trials (RCTs), a statistically significant positive impact of QTR PD has been demonstrated on teaching practices and student outcomes. A further question that remains is whether the change in teaching quality through involvement in QTR PD improves student achievement. This underlying mechanism is yet to be explored in the QTR PD framework and has rarely been explored in PD settings internationally due to limited data that often prevents suitable statistical techniques to be employed. A clustered RCT investigating the impact of QTR PD on teaching quality and student achievement was conducted in 2019, involving 133 government primary schools in New South Wales (NSW), Australia. Data were collected from 222 teachers and 5146 students in Stage 2 (years 3-4) in Term 1 and these teachers and students were followed up in Term 4 of the same year. Teaching practices were observed and rated in the classroom using a pedagogical model, the Quality Teaching (QT) model, which contains 18 observable elements within three dimensions of teaching practice (Intellectual Quality, Quality Learning Environment and Significance), while students were assessed using the Progressive Achievement Test (PAT) in mathematics, which was measured using the scaled scores on 40 multiple choice questions. The data structure therefore comprised a combination of multilevel and longitudinal features along with latent constructs and multiple intervention groups (QTR vs wait-list control) that were being compared. This paper examines the underlying interconnected relationships between PD, teaching quality and student achievement by testing the hypothesis that the impacts of the QTR intervention on student achievement in mathematics was mediated by teaching quality. Multilevel structural equation modelling (MSEM) with 2-2-1 design is investigated for these data. Student PAT scores in mathematics were significantly higher, on average (0.11SD [95% CI = 0.01,0.20]) in the intervention group (QTR) compared to those in the control group for the Intellectual Quality (IQ) dimension of the QT model. This demonstrates the statistically significant mediation effect of Intellectual Quality (IQ) on student learning outcomes.