Revisiting generalizability theory in the age of artificial intelligence: Implications for empirical educational research

Abstract

The rise of AI in education presents both transformative opportunities and methodological challenges. This paper revisits Generalizability Theory (G-Theory) as a robust framework to assess the reliability and fairness of AI-driven tools across diverse educational contexts. It is argued that G-Theory’s variance decomposition logic is uniquely suited to disentangle the multifaceted sources of error introduced by evolving AI systems, user diversity, and complex learning environments. Through empirical use cases it is illustrated how G-Theory can support the design of equitable, scalable, and context-sensitive AI applications. We further A G-Theory Readiness Checklist to guide researchers in designing studies with AI as a methodological facet is proposed. Finally, conceptual, technical, ethical, pedagogical, and regulatory limitations and implications for study designs are highlighted. The paper concludes with suggestions for future research.