Lost in Translation: Determining the Generalizability of Temporal Models across Course Contexts

A common activity in learning analytics research is to demonstrate a new analytical technique by applying it to data from a single course. We explore whether the value of an analytical approach might generalize across course contexts. Accordingly, we conduct a conceptual replication of a well-cited temporal modeling study using self-regulated learning (SRL) taxonomies. We attempt to conceptually replicate this previous work through the analysis of 411 students across 19 courses’ trace event data. Using established SRL categorizations, learner actions are sequenced to identify regular clusters of interaction through hierarchical clustering methods. These clusters are then compared with the entire data corpus and each other through the development of first-order Markov models to develop process maps. Our findings indicate that, although some general patterns of SRL can generalize, these results are more limited at higher scales. Comparing these clusters of interaction along students’ performance in courses also indicates some relationships between activity and outcomes, though this finding is also limited in relation to the complexity introduced by scaling out these methods. We discuss how these temporal models should be viewed when making descriptive and qualitative inferences about students’ activity in digital learning environments.