Decomposition of WAIC for assessing the information gain with application to educational testing.

Abstract

Nowadays, multidimensional data are often available from educational testing. One natural issue is to identify whether more dimensional data are useful in fitting the item response data. To address this important issue, we develop a new decomposition of Widely Applicable Information Criterion (WAIC) via the posterior predictive ordinate (PPO) under the joint model for the response, response time and two additional educational testing scores. Based on this decomposition, a new model assessment criterion is then proposed, which allows us to determine which of the response time and two additional scores are most useful in fitting the response data and whether other dimensional data are further needed given that one of these dimensional data is already included in the joint model with the response data. In addition, an efficient Monte Carlo method is developed to compute PPO. An extensive simulation study is conducted to examine the empirical performance of the proposed joint model and the model assessment criterion in the psychological setting. The proposed methodology is further applied to an analysis of a real dataset from a computerized educational assessment program.