Leveraging dynamic few-shot prompting and ensemble method for task-oriented dialogue with subjective knowledge

Abstract

Subjective knowledge is key to meeting customer needs. Thus, the Subjective Knowledge-grounded Task-oriented Dialogue (SK-TOD) task tries to accommodate subjective user requests like “Does the restaurant have a good atmosphere?” by choosing relevant subjective knowledge snippets and generating appropriate responses. However, unlike existing methods like retrieval-augmented generation using external objective knowledge, selecting subjective knowledge and summarizing opinions from reviews in a specified scope pose new challenges. Therefore, this paper proposes the DESIGN (Dynamic fEw-Shot promptInG and eNsemble) method for SK-TOD. Specifically, DESIGN first adopts Aspect-Based Sentiment Analysis (ABSA) to enhance subjective knowledge snippets and then builds an ensemble composed of diverse base models for knowledge selection (KS). Here, the base models include both classification models and generative models. At last, for response generation (RG), DESIGN employs generative models conditioned on dialogue context and ABSA-enhanced knowledge. Particularly, we devise the sample selection via the similarity-alignment algorithm to choose similar samples dynamically for the few-shot prompting of KS and RG. We experiment on the 11th Dialog System Technology Challenge (DSTC11) SK-TOD benchmark and an extended dataset, ReDial, with 6147 instances. For KS, we beat the winner of DSTC11 and boosted the F1 for 7% regarding the baseline and achieved 86.16%. For RG, DESIGN outperforms baselines and the DSTC11 winner across eight metrics.E.g., DESIGN improves entailment performance by 5% over the DSTC11 winner and 10% over the baseline.¹