A statistical approach for estimating user satisfaction in spoken human-machine interaction

Abstract

This paper addresses a new approach for statistical modeling of user satisfaction in Spoken Dialogue Systems (SDS) and thereby allows an online monitoring of spoken human-machine interaction. The presented technique relies on a large set of input variables originating from system log files that quantify the ongoing spoken human-machine interaction. The target variable, user satisfaction (US), is captured in a lab study on a 5 point scale with 46 users interacting with an SDS. The model, which is based on Support Vector Machines (SVM) yields a performance of 49.2% unweighted average recall (Cohen's κ = .442, Spearman's ρ = .668) and significantly outperforms related work in that field.