A Suite for Acoustic Language Model Evaluation

Gallil Maimon, Amit Roth, Yossi Adi
{"title":"A Suite for Acoustic Language Model Evaluation","authors":"Gallil Maimon, Amit Roth, Yossi Adi","doi":"arxiv-2409.07437","DOIUrl":null,"url":null,"abstract":"Speech language models have recently demonstrated great potential as\nuniversal speech processing systems. Such models have the ability to model the\nrich acoustic information existing in audio signals, beyond spoken content,\nsuch as emotion, background noise, etc. Despite this, evaluation benchmarks\nwhich evaluate awareness to a wide range of acoustic aspects, are lacking. To\nhelp bridge this gap, we introduce SALMon, a novel evaluation suite\nencompassing background noise, emotion, speaker identity and room impulse\nresponse. The proposed benchmarks both evaluate the consistency of the\ninspected element and how much it matches the spoken text. We follow a\nmodelling based approach, measuring whether a model gives correct samples\nhigher scores than incorrect ones. This approach makes the benchmark fast to\ncompute even for large models. We evaluated several speech language models on\nSALMon, thus highlighting the strengths and weaknesses of each evaluated\nmethod. Code and data are publicly available at\nhttps://pages.cs.huji.ac.il/adiyoss-lab/salmon/ .","PeriodicalId":501284,"journal":{"name":"arXiv - EE - Audio and Speech Processing","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - EE - Audio and Speech Processing","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.07437","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

Speech language models have recently demonstrated great potential as universal speech processing systems. Such models have the ability to model the rich acoustic information existing in audio signals, beyond spoken content, such as emotion, background noise, etc. Despite this, evaluation benchmarks which evaluate awareness to a wide range of acoustic aspects, are lacking. To help bridge this gap, we introduce SALMon, a novel evaluation suite encompassing background noise, emotion, speaker identity and room impulse response. The proposed benchmarks both evaluate the consistency of the inspected element and how much it matches the spoken text. We follow a modelling based approach, measuring whether a model gives correct samples higher scores than incorrect ones. This approach makes the benchmark fast to compute even for large models. We evaluated several speech language models on SALMon, thus highlighting the strengths and weaknesses of each evaluated method. Code and data are publicly available at https://pages.cs.huji.ac.il/adiyoss-lab/salmon/ .
声学语言模型评估套件
最近,语音语言模型作为通用语音处理系统已显示出巨大的潜力。这些模型有能力对音频信号中存在的丰富声学信息进行建模,而不局限于语音内容,如情感、背景噪声等。尽管如此,目前还缺乏对广泛的声学方面进行评估的基准。为了填补这一空白,我们推出了 SALMon,这是一个新颖的评估套件,包含背景噪声、情感、说话者身份和房间脉冲响应。所提出的基准既能评估检测元素的一致性,也能评估其与口语文本的匹配程度。我们采用的是基于建模的方法,衡量模型给出的正确样本得分是否高于错误样本。这种方法即使对于大型模型也能快速计算基准。我们在 SALMon 上评估了几种语音语言模型,从而突出了每种评估方法的优缺点。代码和数据可在https://pages.cs.huji.ac.il/adiyoss-lab/salmon/。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信