SSR-Speech: Towards Stable, Safe and Robust Zero-shot Text-based Speech Editing and Synthesis

Helin Wang, Meng Yu, Jiarui Hai, Chen Chen, Yuchen Hu, Rilin Chen, Najim Dehak, Dong Yu
{"title":"SSR-Speech: Towards Stable, Safe and Robust Zero-shot Text-based Speech Editing and Synthesis","authors":"Helin Wang, Meng Yu, Jiarui Hai, Chen Chen, Yuchen Hu, Rilin Chen, Najim Dehak, Dong Yu","doi":"arxiv-2409.07556","DOIUrl":null,"url":null,"abstract":"In this paper, we introduce SSR-Speech, a neural codec autoregressive model\ndesigned for stable, safe, and robust zero-shot text-based speech editing and\ntext-to-speech synthesis. SSR-Speech is built on a Transformer decoder and\nincorporates classifier-free guidance to enhance the stability of the\ngeneration process. A watermark Encodec is proposed to embed frame-level\nwatermarks into the edited regions of the speech so that which parts were\nedited can be detected. In addition, the waveform reconstruction leverages the\noriginal unedited speech segments, providing superior recovery compared to the\nEncodec model. Our approach achieves the state-of-the-art performance in the\nRealEdit speech editing task and the LibriTTS text-to-speech task, surpassing\nprevious methods. Furthermore, SSR-Speech excels in multi-span speech editing\nand also demonstrates remarkable robustness to background sounds. Source code\nand demos are released.","PeriodicalId":501284,"journal":{"name":"arXiv - EE - Audio and Speech Processing","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-09-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - EE - Audio and Speech Processing","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.07556","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

In this paper, we introduce SSR-Speech, a neural codec autoregressive model designed for stable, safe, and robust zero-shot text-based speech editing and text-to-speech synthesis. SSR-Speech is built on a Transformer decoder and incorporates classifier-free guidance to enhance the stability of the generation process. A watermark Encodec is proposed to embed frame-level watermarks into the edited regions of the speech so that which parts were edited can be detected. In addition, the waveform reconstruction leverages the original unedited speech segments, providing superior recovery compared to the Encodec model. Our approach achieves the state-of-the-art performance in the RealEdit speech editing task and the LibriTTS text-to-speech task, surpassing previous methods. Furthermore, SSR-Speech excels in multi-span speech editing and also demonstrates remarkable robustness to background sounds. Source code and demos are released.
SSR-Speech:实现基于文本的稳定、安全和鲁棒的零镜头语音编辑与合成
本文介绍了 SSR-Speech,它是一种神经编解码器自回归模型,专为稳定、安全和鲁棒的零镜头文本语音编辑和文本到语音合成而设计。SSR-Speech 建立在变换器解码器的基础上,并加入了无分类器引导,以增强生成过程的稳定性。我们还提出了一种水印编码器,用于在语音的编辑区域嵌入帧级水印,以便检测哪些部分被编辑。此外,波形重建利用了未经编辑的原始语音片段,与编码解码模型相比,恢复效果更佳。我们的方法在 RealEdit 语音编辑任务和 LibriTTS 文本到语音任务中达到了最先进的性能,超过了以前的方法。此外,SSR-Speech 在多跨度语音编辑方面表现出色,对背景声音也表现出显著的鲁棒性。源代码和演示版已发布。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信