DA-BERT: Enhancing Knowledge Selection in Dialog via Domain Adapted BERT with Dynamic Masking Probability

Abstract

One of the most challenging tasks in Knowledge-grounded Task-oriented Dialog Systems (KTDS) is the knowledge selection task, which aims to find the proper knowledge snippets to handle user requests. This paper proposes DA-BERT to employ pre-trained BERT with domain adaptive training and newly proposed dynamic masking probability to deal with knowledge selection in KTDS. Domain adaptive training minimizes the domain gap between the general text data BERT is pre-trained on and the dialog-knowledge joint data; and dynamic masking probability enhances the training in an easy-to-hard mode. Experimental results on the benchmark dataset show that our proposed training method outperforms the state-of-the-art models with large margins across all the evaluation metrics. Moreover, we analyze the bad case of our method and recognize several typical errors in the bad case set to facilitate further research in this direction.