A Comparison of Common IRT Model-selection Methods with Mixed-Format Tests

Abstract

ABSTRACT To date, only frequentist model-selection methods have been studied with mixed-format data in the context of IRT model-selection, and it is unknown how popular Bayesian model-selection methods such as DIC, WAIC, and LOO perform. In this study, we present the results of a comprehensive simulation study that compared the performances of eight model-selection methods with mixed-format data to select the correct combination of IRT models. Findings of the simulation study indicate that DIC, WAIC, and LOO had excellent statistical power to choose the correct IRT model combination. They performed comparably with LRT and slightly preferably than AIC, and considerably better than BIC, AICc, and SABIC. In addition, the performances of the three Bayesian methods were more stable than those of AIC and LRT regardless of the sample size and ability distribution. The eight model-selection methods were applied to a real dataset for demonstration purpose.