A robust dialogue evaluation metric exploiting denoising, pre-training and ensembling

Abstract

Robustness plays a vital role in dialogue evaluation, so the 11th Dialogue System Technology Challenge track 4 task 2 (DSTC11.T4.2for short) expects novel robust metrics for paraphrased/back-translated English sentences. However, there are at least three reasons why the competition results are unsatisfactory. First, denoising modules are missing. Second, the proposed metrics rely too much on off-the-shelf models. Third, grammatical correctness is overlooked. Therefore, we propose a novel approach REOPEN (Robust dialogue Evaluation metric exploiting denOising, Pre-training and ENsembling). There are three pioneering ideas. First, REOPENintroduces a new denoising module based on paragraph restoration. Second, REOPENtrains a robust model from scratch by utilizing masked language modeling and semantic matching tasks. Third, REOPENprovides a specific grammatical model that can exclude flaws by paraphrasing. Our experiments substantiate the effectiveness and superiority of REOPENon the DSTC11.T4.2test set and an extended DailyDialog dataset. E.g., the average turn-level Spearman coefficients between REOPENand human judgments are 0.6934, while the corresponding value of the winner of DSTC11.T4.2is 0.4890¹.