M2R-Whisper:多阶段、多尺度检索增强技术,用于增强耳语功能

Jiaming Zhou, Shiwan Zhao, Jiabei He, Hui Wang, Wenjia Zeng, Yong Chen, Haoqin Sun, Aobo Kong, Yong Qin
{"title":"M2R-Whisper:多阶段、多尺度检索增强技术,用于增强耳语功能","authors":"Jiaming Zhou, Shiwan Zhao, Jiabei He, Hui Wang, Wenjia Zeng, Yong Chen, Haoqin Sun, Aobo Kong, Yong Qin","doi":"arxiv-2409.11889","DOIUrl":null,"url":null,"abstract":"State-of-the-art models like OpenAI's Whisper exhibit strong performance in\nmultilingual automatic speech recognition (ASR), but they still face challenges\nin accurately recognizing diverse subdialects. In this paper, we propose\nM2R-whisper, a novel multi-stage and multi-scale retrieval augmentation\napproach designed to enhance ASR performance in low-resource settings. Building\non the principles of in-context learning (ICL) and retrieval-augmented\ntechniques, our method employs sentence-level ICL in the pre-processing stage\nto harness contextual information, while integrating token-level k-Nearest\nNeighbors (kNN) retrieval as a post-processing step to further refine the final\noutput distribution. By synergistically combining sentence-level and\ntoken-level retrieval strategies, M2R-whisper effectively mitigates various\ntypes of recognition errors. Experiments conducted on Mandarin and subdialect\ndatasets, including AISHELL-1 and KeSpeech, demonstrate substantial\nimprovements in ASR accuracy, all achieved without any parameter updates.","PeriodicalId":501284,"journal":{"name":"arXiv - EE - Audio and Speech Processing","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"M2R-Whisper: Multi-stage and Multi-scale Retrieval Augmentation for Enhancing Whisper\",\"authors\":\"Jiaming Zhou, Shiwan Zhao, Jiabei He, Hui Wang, Wenjia Zeng, Yong Chen, Haoqin Sun, Aobo Kong, Yong Qin\",\"doi\":\"arxiv-2409.11889\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"State-of-the-art models like OpenAI's Whisper exhibit strong performance in\\nmultilingual automatic speech recognition (ASR), but they still face challenges\\nin accurately recognizing diverse subdialects. In this paper, we propose\\nM2R-whisper, a novel multi-stage and multi-scale retrieval augmentation\\napproach designed to enhance ASR performance in low-resource settings. Building\\non the principles of in-context learning (ICL) and retrieval-augmented\\ntechniques, our method employs sentence-level ICL in the pre-processing stage\\nto harness contextual information, while integrating token-level k-Nearest\\nNeighbors (kNN) retrieval as a post-processing step to further refine the final\\noutput distribution. By synergistically combining sentence-level and\\ntoken-level retrieval strategies, M2R-whisper effectively mitigates various\\ntypes of recognition errors. Experiments conducted on Mandarin and subdialect\\ndatasets, including AISHELL-1 and KeSpeech, demonstrate substantial\\nimprovements in ASR accuracy, all achieved without any parameter updates.\",\"PeriodicalId\":501284,\"journal\":{\"name\":\"arXiv - EE - Audio and Speech Processing\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-09-18\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - EE - Audio and Speech Processing\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2409.11889\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - EE - Audio and Speech Processing","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.11889","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

摘要

OpenAI 的 Whisper 等最先进的模型在多语种自动语音识别(ASR)方面表现出强劲的性能,但在准确识别不同的子方言方面仍面临挑战。在本文中,我们提出了M2R-whisper,这是一种新颖的多阶段、多尺度检索增强方法,旨在提高低资源环境下的自动语音识别性能。基于上下文学习(ICL)和检索增强技术的原理,我们的方法在预处理阶段采用句子级 ICL 来利用上下文信息,同时将标记级 k-最近邻(kNN)检索整合为后处理步骤,以进一步完善最终输出分布。通过协同结合句子级和标记级检索策略,M2R-whisper 有效地减少了各种类型的识别错误。在普通话和亚方言数据集(包括 AISHELL-1 和 KeSpeech)上进行的实验表明,M2R-whisper 的 ASR 准确率大幅提高,而这一切都无需任何参数更新。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
M2R-Whisper: Multi-stage and Multi-scale Retrieval Augmentation for Enhancing Whisper
State-of-the-art models like OpenAI's Whisper exhibit strong performance in multilingual automatic speech recognition (ASR), but they still face challenges in accurately recognizing diverse subdialects. In this paper, we propose M2R-whisper, a novel multi-stage and multi-scale retrieval augmentation approach designed to enhance ASR performance in low-resource settings. Building on the principles of in-context learning (ICL) and retrieval-augmented techniques, our method employs sentence-level ICL in the pre-processing stage to harness contextual information, while integrating token-level k-Nearest Neighbors (kNN) retrieval as a post-processing step to further refine the final output distribution. By synergistically combining sentence-level and token-level retrieval strategies, M2R-whisper effectively mitigates various types of recognition errors. Experiments conducted on Mandarin and subdialect datasets, including AISHELL-1 and KeSpeech, demonstrate substantial improvements in ASR accuracy, all achieved without any parameter updates.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信