An open concept metric for assessing dialog system complexity

Abstract

Techniques for assessing dialog system performance commonly focus on characteristics of the interaction, using metrics such as completion, satisfaction or time on task. However, such metrics are not always capable of differentiating systems that operate on fundamentally different principles, particularly when tested on tasks that focus on common-denominator capabilities. We introduce a new metric, the open concept count, and show how it can be used to capture useful system properties of a dialog system.