A process-oriented perspective on pre-service teachers' self-efficacy and their motivational messages: Using large language models to classify teachers' speech.

Abstract

Background: Recent studies have examined the relation between teacher motivation, motivational messages and student learning but are limited to an achievement-related context, primarily using survey data. Moreover, our understanding of the relation between various teacher characteristics, such as teacher self-efficacy (TSE), and their motivational message use remains limited.

Aims: Our study tested whether teacher speech can be classified into self-determination (SDT)-based motivational messages and reliably assessed with a large language model (LLM). Additionally, we analysed the relation between pre-service TSE and their motivational message use.

Sample: For our first aim, we used human-rater annotations from 119 pre-service teachers' classroom recordings. For our second aim, we used data from 103 pre-service teachers (52.69% female; M_age = 22.98, SD_age = 3.26, Min_age = 19, Max_age = 34) who participated in a survey and were video-recorded while teaching.

Methods: First, we manually classified pre-service teachers' motivational messages based on transcripts and used human-rater annotations to fine-tune an LLM. Second, we analysed the relation between pre-service TSE and motivational message use.

Results and conclusions: The fine-tuned LLM demonstrated promising performance in assessing SDT-based motivational messages but needs further refining to assess thwarting messages. The analysis with human annotation showed that pre-service TSE for classroom management positively affected the frequency of relatedness-supportive messages. Pre-service TSE for student engagement increased the likelihood of never using a competence- or relatedness-thwarting message. Pre-service TSE for instructional strategies reduced the frequency of autonomy-supportive messages. LLM-based analyses showed slightly different results but did not contradict human annotation-based analyses.