Investigating the effect of transformer encoder architecture to improve the reliability of classroom observation ratings on high-inference discourse.

Abstract

This study investigates the effect of transformer encoder architecture on the classification accuracy of high-inference discourse elements in classroom settings. Recognizing the importance of capturing nuanced interactions between students and teachers, our study explores the performance of different transformer models, focusing particularly on the bi-encoder architecture of S-BERT. We evaluated various embedding strategies, along with different pooling methods, to optimize the bi-encoder model's classification accuracy for discourse elements such as High Uptake and Focusing Question. We compared S-BERT's performance with traditional cross-encoding transformer models such as BERT and RoBERTa. Our results demonstrate that S-BERT, particularly with a batch size of 8, learning rate of 2e-5, and specific embedding strategies, significantly outperforms other baseline models, achieving F1 scores up to 0.826 for High Uptake and 0.908 for Focusing Question. Our findings highlighted the importance of customized vectorization strategies, encompassing individual and interaction features (dot-product and absolute distance), and underscores the need to carefully select pooling methods to enhance performance. Our findings offer valuable insights into the design of transformer models for classroom discourse analysis, contributing to the advancement of NLP methods in educational research.