Generalizability Theory Approach to Analyzing Automated-Item Generated Test Forms

Abstract

This study presents several multivariate Generalizability theory designs for analyzing automatic item-generated (AIG) based test forms. The study used real data to illustrate the analysis procedure and discuss practical considerations. We collected the data from two groups of students, each group receiving a different form generated by AIG. A total of 74 students participated in this study and responded to AIG-based test forms. Then, we analyzed the data using four distinct designs based on the data collection design, and conceptualization of true scores and measurement conditions over hypothetical replications. This study also examined the theoretical relationships among the four data collection designs and highlighted the potential impact of confounding between item templates and item clones.