Gaurav Maheshwari, Dmitry Ivanov, Théo Johannet, Kevin El Haddad
{"title":"ASR 基准测试:需要更具代表性的对话数据集","authors":"Gaurav Maheshwari, Dmitry Ivanov, Théo Johannet, Kevin El Haddad","doi":"arxiv-2409.12042","DOIUrl":null,"url":null,"abstract":"Automatic Speech Recognition (ASR) systems have achieved remarkable\nperformance on widely used benchmarks such as LibriSpeech and Fleurs. However,\nthese benchmarks do not adequately reflect the complexities of real-world\nconversational environments, where speech is often unstructured and contains\ndisfluencies such as pauses, interruptions, and diverse accents. In this study,\nwe introduce a multilingual conversational dataset, derived from TalkBank,\nconsisting of unstructured phone conversation between adults. Our results show\na significant performance drop across various state-of-the-art ASR models when\ntested in conversational settings. Furthermore, we observe a correlation\nbetween Word Error Rate and the presence of speech disfluencies, highlighting\nthe critical need for more realistic, conversational ASR benchmarks.","PeriodicalId":501284,"journal":{"name":"arXiv - EE - Audio and Speech Processing","volume":"5 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"ASR Benchmarking: Need for a More Representative Conversational Dataset\",\"authors\":\"Gaurav Maheshwari, Dmitry Ivanov, Théo Johannet, Kevin El Haddad\",\"doi\":\"arxiv-2409.12042\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Automatic Speech Recognition (ASR) systems have achieved remarkable\\nperformance on widely used benchmarks such as LibriSpeech and Fleurs. However,\\nthese benchmarks do not adequately reflect the complexities of real-world\\nconversational environments, where speech is often unstructured and contains\\ndisfluencies such as pauses, interruptions, and diverse accents. In this study,\\nwe introduce a multilingual conversational dataset, derived from TalkBank,\\nconsisting of unstructured phone conversation between adults. Our results show\\na significant performance drop across various state-of-the-art ASR models when\\ntested in conversational settings. Furthermore, we observe a correlation\\nbetween Word Error Rate and the presence of speech disfluencies, highlighting\\nthe critical need for more realistic, conversational ASR benchmarks.\",\"PeriodicalId\":501284,\"journal\":{\"name\":\"arXiv - EE - Audio and Speech Processing\",\"volume\":\"5 1\",\"pages\":\"\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-09-18\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - EE - Audio and Speech Processing\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2409.12042\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - EE - Audio and Speech Processing","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.12042","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
摘要
自动语音识别(ASR)系统在 LibriSpeech 和 Fleurs 等广泛使用的基准测试中表现出色。然而,这些基准并不能充分反映真实世界对话环境的复杂性,因为对话环境中的语音通常是非结构化的,并包含停顿、中断和不同口音等不流畅现象。在这项研究中,我们引入了一个多语言会话数据集,该数据集来自 TalkBank,由成人之间的非结构化电话会话组成。我们的研究结果表明,在会话环境中进行测试时,各种最先进的 ASR 模型的性能明显下降。此外,我们还观察到单词错误率(Word Error Rate)与语音不流畅(speech disfluencies)之间存在相关性,这凸显了对更真实的会话式 ASR 基准的迫切需求。
ASR Benchmarking: Need for a More Representative Conversational Dataset
Automatic Speech Recognition (ASR) systems have achieved remarkable
performance on widely used benchmarks such as LibriSpeech and Fleurs. However,
these benchmarks do not adequately reflect the complexities of real-world
conversational environments, where speech is often unstructured and contains
disfluencies such as pauses, interruptions, and diverse accents. In this study,
we introduce a multilingual conversational dataset, derived from TalkBank,
consisting of unstructured phone conversation between adults. Our results show
a significant performance drop across various state-of-the-art ASR models when
tested in conversational settings. Furthermore, we observe a correlation
between Word Error Rate and the presence of speech disfluencies, highlighting
the critical need for more realistic, conversational ASR benchmarks.