DFM: Dialogue foundation model for universal large-scale dialogue-oriented task learning

Abstract

Building a universal conversational agent has been a long-standing goal of the dialogue research community. Most previous works only focus on a small set of dialogue tasks. In this work, we aim to build a unified dialogue foundation model (DFM) which can be used to solve massive diverse dialogue tasks. To achieve this goal, a large-scale well-annotated dialogue dataset with rich task diversity (DialogZoo) is collected. We introduce a framework to unify all dialogue tasks and propose novel auxiliary self-supervised tasks to achieve stable training of DFM on the highly diverse large scale DialogZoo corpus. Experiments show that, compared with models of the same size, DFM can achieve competitive performance on very rich cross-domain downstream dialogue tasks. Furthermore, when scaling to large language models, DFM remains effective. This demonstrates that DFM largely extends the ability of unified dialogue pre-trained model.