SIGDIAL Conferences最新文献

筛选

英文中文

Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation 您的目标导向对话模型真的表现出色吗？系统评价的经验分析

SIGDIAL Conferences Pub Date : 2020-05-15 DOI: 10.18653/v1/2020.sigdial-1.37

Ryuichi Takanobu, Qi Zhu, Jinchao Li, Baolin Peng, Jianfeng Gao, Minlie Huang

{"title":"Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation","authors":"Ryuichi Takanobu, Qi Zhu, Jinchao Li, Baolin Peng, Jianfeng Gao, Minlie Huang","doi":"10.18653/v1/2020.sigdial-1.37","DOIUrl":"https://doi.org/10.18653/v1/2020.sigdial-1.37","url":null,"abstract":"There is a growing interest in developing goal-oriented dialog systems which serve users in accomplishing complex tasks through multi-turn conversations. Although many methods are devised to evaluate and improve the performance of individual dialog components, there is a lack of comprehensive empirical study on how different components contribute to the overall performance of a dialog system. In this paper, we perform a system-wise evaluation and present an empirical analysis on different types of dialog systems which are composed of different modules in different settings. Our results show that (1) a pipeline dialog system trained using fine-grained supervision signals at different component levels often obtains better performance than the systems that use joint or end-to-end models trained on coarse-grained labels, (2) component-wise, single-turn evaluation results are not always consistent with the overall performance of a dialog system, and (3) despite the discrepancy between simulators and human users, simulated evaluation is still a valid alternative to the costly human evaluation especially in the early stage of development.","PeriodicalId":512520,"journal":{"name":"SIGDIAL Conferences","volume":"23 3","pages":"297-310"},"PeriodicalIF":0.0,"publicationDate":"2020-05-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141204787","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 37

ConvoKit: A Toolkit for the Analysis of Conversations ConvoKit：对话分析工具包

SIGDIAL Conferences Pub Date : 2020-05-08 DOI: 10.18653/v1/2020.sigdial-1.8

Jonathan P. Chang, Caleb Chiam, Liye Fu, Andrew Z. Wang, Justine Zhang, Cristian Danescu-Niculescu-Mizil

引用次数: 71