EXPLANATION-BASED AUTOMATED ASSESSMENT OF OPEN ENDED LEARNER RESPONSES

Abstract

Open ended assessment items require students to freely articulate their thinking as opposed to, for instance, multiple choice questions. Such free generation of answers by students enables what we may call true assessment because these answers offer a direct view of learners’ mental models. Nevertheless, assessing open ended learner responses is extremely challenging, e.g., if done manually by experts it becomes prohibitively expensive to scale up to millions of learners. To address this scalability challenge, automated methods to assess students' free responses are being explored. To this end, we present a novel solution to automatically assess open ended learner responses based on recent advances in computational linguistics and optimization algorithms. Our proposed solution accounts for linguistic phenomena such as anaphora resolution and negation in order to reach a deeper level of semantic interpretation of student answers. This is a key advantage compared to previous methods that focus primarily on distributional semantic representations of texts. Furthermore, our method provides both a holistic score as well as a detailed explanation of the score by performing a concept-level analysis of student responses. We present results obtained with the proposed method on a dataset that is widely used to evaluate automated methods for assessing open ended learner responses. The results indicate that our method is extremely competitive or surpasses the performance of previously proposed methods. Furthermore, by being able to pick on concepts students have yet to articulate, it enables the development of more personalized and dynamic generation of feedback in intelligent tutoring systems.