Semantic Matching Evaluation of User Responses to Electronics Questions in AutoTutor

Relatedness between user input and an ideal response is a salient feature required for proper functioning of an Intelligent Tutoring System (ITS) using natural language processing. Improper assessment of text input causes maladaptation in ITSs. Meta-assessment of user responses in ITSs can improve instruction efficacy and user satisfaction. Therefore, this paper evaluates the quality of semantic matching between user input and the expected response in AutoTutor, an ITS which holds a conversation with the user in natural language. AutoTutor's dialogue is driven by the AutoTutor Conversation Engine (ACE), which uses a combination of Latent Semantic Analysis (LSA) and Regular Expressions (RegEx) to assess user input. We assessed ACE via responses from 219 Amazon Mechanical Turk users, who answered 118 electronics questions broken into 5202 response pairings (n = 5202). These analyses explore the relationship between RegEx and LSA, agreement between the two judges, and agreement between human judges and ACE. Additionally, we calculated precision and recall. As expected, regular expressions and LSA had a moderate, positive relationship, and the agreement between ACE and human was fair, but slightly lower than agreement between human.