Post Processing Selection of Automatic Item Generation in Testing to Ensure Human-Like Quality with Machine Learning

Abstract

Automatic Item Generation (AIG) is increasingly used to process large amounts of information and scale the demand for computerized testing. Recent work in Artificial Intelligence for AIG (aka Natural Question Generation-NQG), states that even newer AIG techniques are short in syntactic, semantic, and contextual relevance when evaluated qualitatively on small datasets. We confirm this deficiency quantitatively over large datasets. Additionally, we find that human evaluation by Subject Matter Experts (SMEs) conservatively rejects at least ∼9% portion of AI test questions in our experiment over large diverse dataset topics. Here we present an analytical study of these differences, and this motivates our two-phased post-processing AI daisy chain machine learning (ML) architecture for selection and editing of AI generated questions using current techniques. Finally, we identify and propose the first selection step in the daisy chain using ML with 97+% accuracy, and provide analytical guidance for development of the second editing step with a measured lower bound on a BLEU score improvement of 2.4+%.