Bo Liu, Liming Zhan, Yujie Feng, Zexin Lu, Chengqiang Xie, Lei Xue, Xiao-Ming Wu, Albert Y. S. Lam
{"title":"用于分布外意图检测的基于多样性的信道原型学习","authors":"Bo Liu, Liming Zhan, Yujie Feng, Zexin Lu, Chengqiang Xie, Lei Xue, Xiao-Ming Wu, Albert Y. S. Lam","doi":"arxiv-2409.11114","DOIUrl":null,"url":null,"abstract":"In the realm of task-oriented dialogue systems, a robust intent detection\nmechanism must effectively handle malformed utterances encountered in\nreal-world scenarios. This study presents a novel fine-tuning framework for\nlarge language models (LLMs) aimed at enhancing in-distribution (ID) intent\nclassification and out-of-distribution (OOD) intent detection, which utilizes\nsemantic matching with prototypes derived from ID class names. By harnessing\nthe highly distinguishable representations of LLMs, we construct semantic\nprototypes for each ID class using a diversity-grounded prompt tuning approach.\nWe rigorously test our framework in a challenging OOD context, where ID and OOD\nclasses are semantically close yet distinct, referred to as \\emph{near} OOD\ndetection. For a thorough assessment, we benchmark our method against the\nprevalent fine-tuning approaches. The experimental findings reveal that our\nmethod demonstrates superior performance in both few-shot ID intent\nclassification and near-OOD intent detection tasks.","PeriodicalId":501030,"journal":{"name":"arXiv - CS - Computation and Language","volume":"32 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Diversity-grounded Channel Prototypical Learning for Out-of-Distribution Intent Detection\",\"authors\":\"Bo Liu, Liming Zhan, Yujie Feng, Zexin Lu, Chengqiang Xie, Lei Xue, Xiao-Ming Wu, Albert Y. S. Lam\",\"doi\":\"arxiv-2409.11114\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"In the realm of task-oriented dialogue systems, a robust intent detection\\nmechanism must effectively handle malformed utterances encountered in\\nreal-world scenarios. This study presents a novel fine-tuning framework for\\nlarge language models (LLMs) aimed at enhancing in-distribution (ID) intent\\nclassification and out-of-distribution (OOD) intent detection, which utilizes\\nsemantic matching with prototypes derived from ID class names. By harnessing\\nthe highly distinguishable representations of LLMs, we construct semantic\\nprototypes for each ID class using a diversity-grounded prompt tuning approach.\\nWe rigorously test our framework in a challenging OOD context, where ID and OOD\\nclasses are semantically close yet distinct, referred to as \\\\emph{near} OOD\\ndetection. For a thorough assessment, we benchmark our method against the\\nprevalent fine-tuning approaches. The experimental findings reveal that our\\nmethod demonstrates superior performance in both few-shot ID intent\\nclassification and near-OOD intent detection tasks.\",\"PeriodicalId\":501030,\"journal\":{\"name\":\"arXiv - CS - Computation and Language\",\"volume\":\"32 1\",\"pages\":\"\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-09-17\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - CS - Computation and Language\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2409.11114\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Computation and Language","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.11114","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
摘要
在面向任务的对话系统领域,强大的意图检测机制必须能有效处理真实世界场景中遇到的畸形语句。本研究为大语言模型(LLMs)提出了一个新颖的微调框架,旨在增强分布内(ID)意图分类和分布外(OOD)意图检测,该框架利用了从 ID 类名衍生出的原型进行语义匹配。我们在具有挑战性的 OOD 环境中对我们的框架进行了严格测试,在这种环境中,ID 和 OOD 类别在语义上非常接近,但又截然不同,这被称为 "接近 "OOD 检测。OODdetection.为了进行全面评估,我们将我们的方法与流行的微调方法进行了比较。实验结果表明,我们的方法在少量 ID 意图分类和近似 OOD 意图检测任务中都表现出了卓越的性能。
Diversity-grounded Channel Prototypical Learning for Out-of-Distribution Intent Detection
In the realm of task-oriented dialogue systems, a robust intent detection
mechanism must effectively handle malformed utterances encountered in
real-world scenarios. This study presents a novel fine-tuning framework for
large language models (LLMs) aimed at enhancing in-distribution (ID) intent
classification and out-of-distribution (OOD) intent detection, which utilizes
semantic matching with prototypes derived from ID class names. By harnessing
the highly distinguishable representations of LLMs, we construct semantic
prototypes for each ID class using a diversity-grounded prompt tuning approach.
We rigorously test our framework in a challenging OOD context, where ID and OOD
classes are semantically close yet distinct, referred to as \emph{near} OOD
detection. For a thorough assessment, we benchmark our method against the
prevalent fine-tuning approaches. The experimental findings reveal that our
method demonstrates superior performance in both few-shot ID intent
classification and near-OOD intent detection tasks.