Methodological foundations for the measurement of learning in learning analytics

Abstract

Learning analysts often claim to measure learning, but their work has attracted growing concern about whether or not the measures are sufficiently accurate, fair, reliable, and valid, with utility for educators and interpretable by them. This paper considers these issues in the light of practices of scholars in more established fields, educational measurement particularly. The focus is on what really matters about methodologies for measuring learning, including foundational assumptions about the nature of learning, what is understood by the term `measured', the criteria applied when assessing quality of data, the standards of proof required to establish validity, reliability, generalizability, utility and interpretability of findings, and assumptions about learners and learning underlying data modeling techniques used to abstract meaning from the data. This paper argues that, for learning analytics to take its place as a fully-fledged member of the learning sciences, it needs seriously to consider how to measure learning. Methodology crafted at the interface of measurement science and learning analytics may be of sufficient interest to create a new subfield of scholarship - dubbed here `metrilytics' - to make a distinctive contribution to the science of learning.