Design Analytics for Mobile Learning: Scaling up the Classification of Learning Designs Based on Cognitive and Contextual Elements

Abstract

This research was triggered by the identified need in literature for large-scale studies about the kinds of designs that teachers create for mobile learning (m-learning). These studies require analyses of large datasets of learning designs. The common approach followed by researchers when analyzing designs has been to manually classify them following high-level pedagogically guided coding strategies, which demands extensive work. Therefore, the first goal of this paper is to explore the use of supervised machine learning (SML) to automatically classify the textual content of m-learning designs using pedagogically relevant classifications, such as the cognitive level demanded by students to carry out specific designed tasks, the phases of inquiry learning represented in the designs, or the role that the situated environment has in the designs. Because not all SML models are transparent, but researchers often need to understand their behaviour, the second goal of this paper is to consider the trade-off between models’ performance and interpretability in the context of design analytics for m-learning. To achieve these goals, we compiled a dataset of designs deployed using two tools, Avastusrada and Smartzoos. With this dataset, we trained and compared different models and feature extraction techniques. We further optimized and compared the best performing and most interpretable algorithms (EstBERT and Logistic Regression) to consider the second goal with an illustrative case. We found that SML can reliably classify designs with accuracy > 0.86 and Cohen’s kappa > 0.69.