Quantitative evaluation of dialog corpora acquired through different techniques

Abstract

In this paper, we present the results of the comparison between three corpora acquired by means of different techniques. The first corpus was acquired using the Wizard of Oz technique. A statistical user simulation technique has been developed for the acquisition of the second corpus. In this technique, the next user answer is selected by means of a classification process that takes into account the previous user turns, the last system answer and the objective of the dialog. Finally, a dialog simulation technique has been developed for the acquisition of the third corpus. This technique uses a random selection of the user and system turns, defining stop conditions for automatically deciding if the simulated dialog is successful or not. We use several evaluation measures proposed in previous research to compare between our three acquired corpora, and then discuss the similarities and differences with regard to these measures.