Using hybrid intelligence to enhance peer feedback for promoting teacher reflection in video-based online learning

Abstract

Video-based teacher online learning enables teachers to engage in reflective practice by watching others' classroom videos, providing peer feedback (PF) and reviewing others' work. However, the quality and reliability of PF often suffer due to variations in teaching proficiency among providers, which limits its usefulness for reviewers. To improve the quality of PF, hybrid intelligence is a promising approach that enhances human evaluation with AI-based techniques. Thus, we developed a hybrid intelligence feedback (HIF) system, where PF is categorized and summarized by large language models (LLM), and accompanied with AI multimodal data analysis, all in accordance with a knowledge structure. To investigate the effectiveness of the HIF, we conducted a study involving 58 pre-service mathematics teachers. After their initial feedback provision on a classroom video, they were divided into two groups. One group received HIF, while the other received traditional PF. Both groups revised their initial feedback based on the same video with PF or HIF report, and were assigned two tasks, namely in-depth reflection and extensive reflection. We analysed the reflective writings generated in in-depth reflection using the Structure of Observed Learning Outcomes taxonomy, and examined the diversity of teachers' attentions in extensive reflection using information entropy. Compared to traditional PF, our findings indicated that HIF (a) facilitated more comments added in feedback revision, (b) promoted multi-structural and extended abstract level thinking in in-depth reflection, (c) encouraged more diverse attentions in extensive reflection. These results demonstrate the effectiveness of HIF in enhancing PF to promote reviewers' reflection. This efficacy can be attributed to the utilization of LLM to identify common elements within PF, guided by the human knowledge-based framework, as well as the integration of data-driven evidence to complement PF.

Practitioner notes

What is already known about this topic?

• Video-based teacher online learning allows teachers to reflect on their own or others' videos flexibly while providing and reviewing peer feedback using reflection tools.
• While the benefits of reflecting on one's own videos with peer feedback are widely recognized, there is limited empirical evidence supporting the advantages of reflecting on others' videos with peer feedback. The effectiveness of this process may be affected by the quality and reliability of the peer feedback provided.
• Using natural language processing techniques to enhance peer feedback can be feasible and effective. However, it is primarily used to address textual-level issues and is less effective in promoting professionalism.
• Multimodal data analysis has shown effectiveness in enhancing teaching behaviours and facilitating reflection. However, despite the widespread use of AI-based analysis on classroom videos, they often lack educational interpretations.
• Hybrid intelligence is a novel concept in learning science research, aiming to leverage both human and machine intelligence to enhance the overall effectiveness.

What this paper adds?

• This study applied the concept of hybrid intelligence to video-based teacher learning by proposing a hybrid intelligence feedback (HIF) system, aiming to promote teachers' reflection on others' videos during the peer feedback review process.
• This study proposed the design of HIF module, where human peer feedback was enhanced by large language models, and machine data analysis was complemented with educational interpretations, all structured according to an expert knowledge structure.
• The results showed that the HIF was effective to stimulate teachers' higher-level thinking in in-depth reflection and enhanced the diversity of their attentions in extensive reflection. However, it may still be challenging for novices to comprehend and integrate newly noticed pedagogical strategies in the HIF with their internal knowledge structures during reflection.

Implications for practice and/or policy

• With the rapid advancement of generative artificial intelligence, the utilization of large language models becomes more flexible and effective, enabling multitasking enhancement for peer feedback in collaboration with human's professional knowledge.
• Multimodal data analysis effectively collaborates with human observations by managing low-level observation aspects, allowing humans to concentrate on higher-level thinking guided by the educational interpretations.
• The effectiveness of the HIF system is influenced by teachers' pedagogical knowledge, prior feedback provision experience and data literacy. In the future research, these diversities need to be taken into account in the design of video-based PD incorporating HIF to assess its long-term efficacy.