Uncertainty-Aware Personalized Readability Assessments for Second Language Learners

Abstract

Assessing whether an ungraded second language learner can read a given text quickly is important for further instructing and supporting the learner, particularly when evaluating numerous ungraded learners from diverse backgrounds. Second language acquisition (SLA) studies have tackled such assessment tasks wherein only a single short vocabulary test result is available to assess a learner; such studies have shown that the text-coverage, i.e., the percentage of words the learner knows in the text, is the key assessment measure. Currently, count-based percentages are used, in which each word in the given text is classified as being known or unknown to the learner, and the words classified as known are then simply counted. When each word is classified, we can also obtain an uncertainty value as to how likely each word is known to the learner. Although such values can be informative for a readability assessment, how to leverage these values to guarantee their use as an assessment measure that is comparable to that of the previous values remains unclear. We propose a novel framework that allows assessment methods to be uncertainty-aware while guaranteeing comparability to the text-coverage threshold. Such methods involve a computationally complex problem, for which we also propose a practical algorithm. In addition, we propose a neural-network based classifier from which we can obtain better uncertainty values. For evaluation, we created a crowdsourcing-based dataset in which a learner takes both vocabulary and readability tests. The best method under our framework outperformed conventional methods.