Hsi-Che Lin, Yi-Cheng Lin, Huang-Cheng Chou, Hung-yi Lee
{"title":"Improving Speech Emotion Recognition in Under-Resourced Languages via Speech-to-Speech Translation with Bootstrapping Data Selection","authors":"Hsi-Che Lin, Yi-Cheng Lin, Huang-Cheng Chou, Hung-yi Lee","doi":"arxiv-2409.10985","DOIUrl":null,"url":null,"abstract":"Speech Emotion Recognition (SER) is a crucial component in developing\ngeneral-purpose AI agents capable of natural human-computer interaction.\nHowever, building robust multilingual SER systems remains challenging due to\nthe scarcity of labeled data in languages other than English and Chinese. In\nthis paper, we propose an approach to enhance SER performance in low SER\nresource languages by leveraging data from high-resource languages.\nSpecifically, we employ expressive Speech-to-Speech translation (S2ST) combined\nwith a novel bootstrapping data selection pipeline to generate labeled data in\nthe target language. Extensive experiments demonstrate that our method is both\neffective and generalizable across different upstream models and languages. Our\nresults suggest that this approach can facilitate the development of more\nscalable and robust multilingual SER systems.","PeriodicalId":501284,"journal":{"name":"arXiv - EE - Audio and Speech Processing","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - EE - Audio and Speech Processing","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.10985","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
Speech Emotion Recognition (SER) is a crucial component in developing
general-purpose AI agents capable of natural human-computer interaction.
However, building robust multilingual SER systems remains challenging due to
the scarcity of labeled data in languages other than English and Chinese. In
this paper, we propose an approach to enhance SER performance in low SER
resource languages by leveraging data from high-resource languages.
Specifically, we employ expressive Speech-to-Speech translation (S2ST) combined
with a novel bootstrapping data selection pipeline to generate labeled data in
the target language. Extensive experiments demonstrate that our method is both
effective and generalizable across different upstream models and languages. Our
results suggest that this approach can facilitate the development of more
scalable and robust multilingual SER systems.
语音情感识别(SER)是开发能够进行自然人机交互的通用人工智能代理的重要组成部分。然而,由于除英语和中文之外的其他语言的标注数据稀缺,构建强大的多语言 SER 系统仍然具有挑战性。在本文中,我们提出了一种通过利用高资源语言的数据来提高低资源语言 SER 性能的方法。具体来说,我们采用了表达式语音到语音翻译(S2ST),并结合新颖的引导数据选择管道来生成目标语言中的标记数据。广泛的实验证明,我们的方法在不同的上游模型和语言中既有效又具有通用性。我们的研究结果表明,这种方法可以促进可扩展性更强的多语言 SER 系统的开发。