2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)最新文献_第6页

Using Self Attention DNNs to Discover Phonemic Features for Audio Deep Fake Detection 基于自关注深度神经网络的语音特征挖掘

2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) Pub Date : 2021-12-13 DOI: 10.1109/ASRU51503.2021.9688312

Hira Dhamyal, Ayesha Ali, I. Qazi, Agha Ali Raza

{"title":"Using Self Attention DNNs to Discover Phonemic Features for Audio Deep Fake Detection","authors":"Hira Dhamyal, Ayesha Ali, I. Qazi, Agha Ali Raza","doi":"10.1109/ASRU51503.2021.9688312","DOIUrl":"https://doi.org/10.1109/ASRU51503.2021.9688312","url":null,"abstract":"With the advancement in natural-sounding speech production models, it is becoming important to develop models that can detect spoofed audios. Synthesized speech models do not explicitly account for all factors affecting speech production, such as the shape, size and structure of a speaker's vocal tract. In this paper, we hypothesize that due to practical limitations of audio corpora (including size, distribution, and balance of variables like gender, age, and accents), there exist certain phonemes that synthesized models are not able to replicate as well as the human articulation system and such phonemes differ in their spectral characteristics from bonafide speech. To discover such phonemes and quantify their effectiveness in distinguishing between spoofed and bonafide speech, we use a deep learning model with self-attention, and analyze the attention weights of the trained model. We use the ASVSpoof2019 dataset for our analysis and find that the attention mechanism picks most on fricatives: /S/,/SH/, nasals: /M/,/N/, vowels: /Y/, and stops: /D/. Furthermore, we obtain 7.54% EER on train and 11.98% on dev data when using only the top-16 most attended phonemes from input audio, better than when any other phoneme classes are used.","PeriodicalId":348315,"journal":{"name":"2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)","volume":"77 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121500548","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 2

Human-Agent Collaboration Strategies for Vision-Grounded Instruction Following 基于视觉指令跟随的人-代理协作策略

2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) Pub Date : 2021-12-13 DOI: 10.1109/ASRU51503.2021.9688143

Guan-Lin Chao, Ian Lane

{"title":"Human-Agent Collaboration Strategies for Vision-Grounded Instruction Following","authors":"Guan-Lin Chao, Ian Lane","doi":"10.1109/ASRU51503.2021.9688143","DOIUrl":"https://doi.org/10.1109/ASRU51503.2021.9688143","url":null,"abstract":"In this paper, we explore two human-agent collaboration strate-gies (human-designed curriculum learning and human-agent dialogue) to boost Reinforcement Learning agent's training in the context of vision-grounded instruction following. The agent is given a text instruction to fetch the described object or switch an appliance. The agent needs to navigate in a simulated multi-room environment, interacts with the objects, furniture, appliances to follow the instruction. Our first strategy is training the agent with a human-designed curriculum. We create a series of subtasks and let the human advisor select several subtasks and design the curriculum composed of the selected tasks in increasing order of complexity. Then the agent is first trained on the human-designed curriculum before training on the target task. Secondly, we enhance the agent with a dialogue module, which allows the agent to query a human collaborator (with a limit number of dialogue exchanges) and obtain three types of hints: action selection, navigation and recognition. Experiments show that the training with a properly designed subtask curriculum can accelerate the performance growth on the target task compared with directly training on the target task, and incorporating a human collaborator dialogue mod-ule further improves the agent's task success rate and route efficiency.","PeriodicalId":348315,"journal":{"name":"2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)","volume":"47 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124071270","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Mandarin Electrolaryngeal Speech Voice Conversion with Sequence-to-Sequence Modeling 用序列到序列建模的普通话电喉语音语音转换

2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) Pub Date : 2021-12-13 DOI: 10.1109/ASRU51503.2021.9687908

Ming-Chi Yen, Wen-Chin Huang, Kazuhiro Kobayashi, Yu-Huai Peng, S. Tsai, Yu Tsao, T. Toda, J. Jang, Hsin-Min Wang

{"title":"Mandarin Electrolaryngeal Speech Voice Conversion with Sequence-to-Sequence Modeling","authors":"Ming-Chi Yen, Wen-Chin Huang, Kazuhiro Kobayashi, Yu-Huai Peng, S. Tsai, Yu Tsao, T. Toda, J. Jang, Hsin-Min Wang","doi":"10.1109/ASRU51503.2021.9687908","DOIUrl":"https://doi.org/10.1109/ASRU51503.2021.9687908","url":null,"abstract":"The electrolaryngeal speech (EL speech) is typically spoken with an electrolarynx device that generates excitation signals to substitute human vocal fold vibrations. Because the excitation signals cannot perfectly characterize sound sources generated by vocal folds, the naturalness and intelligibility of the EL speech are inevitably worse than that of the natural speech (NL speech). To improve speech naturalness, statistical models, such as Gaussian mixture models and deep-learning-based models, have been employed for EL speech voice conversion (ELVC). The ELVC task aims to convert EL speech into NL speech through an ELVC model. To implement a frame-wise ELVC system, accurate feature alignment is crucial for model training. However, the abnormal acoustic characteristics of the EL speech cause misalignments and accordingly limit the ELVC performance. To address this issue, we propose a novel ELVC system based on sequence-to-sequence (seq2seq) modeling with text-to-speech (TTS) pretraining. The seq2seq model involves an attention mechanism to concurrently perform representation learning and alignment. Meanwhile, TTS pretraining provides efficient training with limited data. Experimental results show that the proposed ELVC system yields notable improvements in terms of standardized evaluation metrics and subjective listening tests over a well-known frame-wise ELVC system.","PeriodicalId":348315,"journal":{"name":"2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)","volume":"96 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127142987","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 6

Cyclegean: Cycle Generative Enhanced Adversarial Network for Voice Conversion 循环生成增强的语音转换对抗网络

2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) Pub Date : 2021-12-13 DOI: 10.1109/ASRU51503.2021.9687948

Xulong Zhang, Jianzong Wang, Ning Cheng, Edward Xiao, Jing Xiao

{"title":"Cyclegean: Cycle Generative Enhanced Adversarial Network for Voice Conversion","authors":"Xulong Zhang, Jianzong Wang, Ning Cheng, Edward Xiao, Jing Xiao","doi":"10.1109/ASRU51503.2021.9687948","DOIUrl":"https://doi.org/10.1109/ASRU51503.2021.9687948","url":null,"abstract":"Cycle Generative Adversarial Network (CycleGAN) for voice conversion (VC) task only used discriminators to identify whether the input voice is generated or real. It means the confrontational does not check the similarity with the target voice, leading the generated voice not much similar to the target. In this paper, instead of vocal checking, we propose to enhance the confrontation to target similarity checking that addresses this problem. A Cycle Generative Enhanced Adversarial Network (CycleGEAN) was introduced to make the original two discriminators to target classifier and non-target classifier. The target classifier aims to identify whether the target speaks the input voice or not. Similarly, the non-target classifier identifies the non-target voice. Furthermore, we add a gradient reversal layer with different operations for target and non-target. Then in each GAN, we used both classifiers. One is the discriminator, and the other is trained for using in another GAN. In experiments, the proposed method compare to CycleGAN improves Mean Opinion Score (MOS) of 0.1 and Voice Similarity Score (VSS) of 0.2 on the Voice Conversion Challenge 2018 (VCC2018) dataset.","PeriodicalId":348315,"journal":{"name":"2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)","volume":"67 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114031791","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 11

Audio Embeddings Help to Learn Better Dialogue Policies 音频嵌入有助于学习更好的对话策略

2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) Pub Date : 2021-12-13 DOI: 10.1109/ASRU51503.2021.9688296

Asier López Zorrilla, M. Inés Torres, H. Cuayáhuitl

引用次数: 4

Multi-Granularity Annotation of Instantaneous Intelligibility of Learners' Utterances Based on Shadowing Techniques 基于阴影技术的学习者话语瞬间可理解度多粒度标注

2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) Pub Date : 2021-12-13 DOI: 10.1109/ASRU51503.2021.9688270

Chuanbo Zhu, Ryo Hakoda, D. Saito, N. Minematsu, Noriko Nakanishi, T. Nishimura

{"title":"Multi-Granularity Annotation of Instantaneous Intelligibility of Learners' Utterances Based on Shadowing Techniques","authors":"Chuanbo Zhu, Ryo Hakoda, D. Saito, N. Minematsu, Noriko Nakanishi, T. Nishimura","doi":"10.1109/ASRU51503.2021.9688270","DOIUrl":"https://doi.org/10.1109/ASRU51503.2021.9688270","url":null,"abstract":"The practical goal of pronunciation training is to acquire an intelligible enough pronunciation, not a native-like pronunciation. In our studies [1], [2], we proposed a method that can annotate instantaneous intelligibility of a given L2 English utterance by monitoring listeners' listening behaviors. The listeners were asked to shadow the L2 utterance and the degree of being inarticulate in shadowing was automatically quantified to be used as scores of the instantaneous intelligibility, which were shown to be highly correlated with subjective intelligibility scores. In the present paper, we make an objective assessment of the proposed method, where the automatic scores are compared with those calculated objectively based on manual transcripts of the shadowings. Experiments show that the former scores have such a high correlation as 0.935 with the latter scores, which is higher than correlation obtained using another kind of automatic scores calculated by transcribing the shadowings with ASR. Further, since intelligibility is sometimes discussed in pronunciation training in such smaller units as phonemes, our method is experimentally applied to intelligibility annotation with multiple granularity. Experiments show a high validity of our method to calculate instantaneous intelligibility in units of word, syllable, phoneme, and frame.","PeriodicalId":348315,"journal":{"name":"2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115329842","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 3

Improving Hybrid CTC/Attention End-to-End Speech Recognition with Pretrained Acoustic and Language Models 用预训练的声学和语言模型改进混合CTC/注意力端到端语音识别

2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) Pub Date : 2021-12-13 DOI: 10.1109/ASRU51503.2021.9688009

Keqi Deng, Songjun Cao, Yike Zhang, Long Ma

{"title":"Improving Hybrid CTC/Attention End-to-End Speech Recognition with Pretrained Acoustic and Language Models","authors":"Keqi Deng, Songjun Cao, Yike Zhang, Long Ma","doi":"10.1109/ASRU51503.2021.9688009","DOIUrl":"https://doi.org/10.1109/ASRU51503.2021.9688009","url":null,"abstract":"Recently, self-supervised pretraining has achieved impressive results in end-to-end (E2E) automatic speech recognition (ASR). However, the dominant sequence-to-sequence (S2S) E2E model is still hard to fully utilize the self-supervised pretraining methods because its decoder is conditioned on acoustic representation thus cannot be pretrained separately. In this paper, we propose a pretrained Transformer (Preformer) S2S ASR architecture based on hybrid CTC/attention E2E models to fully utilize the pretrained acoustic models (AMs) and language models (LMs). In our framework, the encoder is initialized with a pretrained AM (wav2vec2.0). The Preformer leverages CTC as an auxiliary task during training and inference. Furthermore, we design a one-cross decoder (OCD), which relaxes the dependence on acoustic representations so that it can be initialized with pretrained LM (DistilGPT2). Experiments are conducted on the AISHELL-1 corpus and achieve a 4.6% character error rate (CER) on the test set. Compared with our vanilla hybrid CTC/attention Transformer baseline, our proposed CTC/attention-based Preformer yields 27% relative CER reduction. To the best of our knowledge, this is the first work to utilize both pretrained AM and LM in a S2S ASR system.","PeriodicalId":348315,"journal":{"name":"2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)","volume":"8 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117181982","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 21

Multi-Stream HiFi-GAN with Data-Driven Waveform Decomposition 数据驱动波形分解的多流HiFi-GAN

2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) Pub Date : 2021-12-13 DOI: 10.1109/ASRU51503.2021.9688194

T. Okamoto, T. Toda, H. Kawai

{"title":"Multi-Stream HiFi-GAN with Data-Driven Waveform Decomposition","authors":"T. Okamoto, T. Toda, H. Kawai","doi":"10.1109/ASRU51503.2021.9688194","DOIUrl":"https://doi.org/10.1109/ASRU51503.2021.9688194","url":null,"abstract":"Although a HiFi-GAN vocoder can synthesize high-fidelity speech waveforms in real time on CPUs, there is a tradeoff between synthesis quality and inference speed. To increase inference speed while maintaining synthesis quality, a multi-band structure is introduced to HiFi-GAN. However, it cannot be trained well because of the strong constraint imposed by the fixed multi-band structure. As an alternative approach, Multi-stream MelGAN and HiFi-GAN are proposed, in which the fixed synthesis filter in Multi-band MelGAN is replaced by a trainable convolutional layer with the same structure. In contrast to Multi-band MelGAN, the proposed methods use the trainable synthesis filter to decompose speech waveforms in a data-driven manner. To evaluate the proposed Multi-stream HiFi-GAN as an entire real-time neural text-to-speech system on CPUs, a fast acoustic model, based on Parallel Tacotron 2 with forced alignment and accentual label input, was implemented. The results of experiments-using Japanese male, female, and multi-speaker corpora-indicate that Multi-stream HiFi-GAN can increase synthesis speed while improving or maintaining synthesis quality in analysis-synthesis and text-to-speech conditions for single-speaker models and unseen speaker synthesis for multi-speaker models, compared with the original HiFi-GAN.","PeriodicalId":348315,"journal":{"name":"2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)","volume":"12 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122395743","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 5

Multitask Generative Adversarial Imitation Learning for Multi-Domain Dialogue System 多领域对话系统的多任务生成对抗模仿学习

2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) Pub Date : 2021-12-13 DOI: 10.1109/ASRU51503.2021.9688234

Chuan-En Hsu, Mahdin Rohmatillah, Jen-Tzung Chien

引用次数: 5

Hearing Faces: Target Speaker Text-to-Speech Synthesis from a Face 听力脸:目标说话人从一张脸的文本到语音合成

2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) Pub Date : 2021-12-13 DOI: 10.1109/ASRU51503.2021.9687866

Björn Plüster, C. Weber, Leyuan Qu, S. Wermter

{"title":"Hearing Faces: Target Speaker Text-to-Speech Synthesis from a Face","authors":"Björn Plüster, C. Weber, Leyuan Qu, S. Wermter","doi":"10.1109/ASRU51503.2021.9687866","DOIUrl":"https://doi.org/10.1109/ASRU51503.2021.9687866","url":null,"abstract":"The existence of a learnable cross-modal association between a person's face and their voice is recently becoming more and more evident. This provides the basis for the task of target speaker text-to-speech (TTS) synthesis from face ref-erence. In this paper, we approach this task by proposing a cross-modal model architecture combining existing unimodal models. We use Tacotron 2 multi-speaker TTS with auditory speaker embeddings based on Global Style Tokens. We trans-fer learn a FaceNet face encoder to predict these embeddings from a static face image reference instead of a voice reference and thus predict a speaker's voice and speaking characteristics from their face. Compared to Face2Speech, the only existing work on this task, we use a more modular architecture that allows the use of openly available and pretrained model components. This approach enables high-quality speech synthesis and allows for an easily extensible model architecture. Ex-perimental results show good matching ability while retaining better voice naturalness than Face2Speech. We examine the limitations of our model and discuss multiple possible av-enues of improvement for future work.","PeriodicalId":348315,"journal":{"name":"2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)","volume":"43 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-12-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131256997","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 3