Investigating multi-task learning for automatic speech recognition with code-switching between mandarin and english

This work investigates a Multi-task Learning (MTL-DNN) approach to enhance the performance of Mandarin-English code-switching conversational speech recognition (MECS-CSR). The approach aims at getting a better acoustic model for the primary task by jointly learning two auxiliary tasks together. To overcome the effect of co-articulation at code-switch points, under MTL-DNN, we propose to jointly train two types of Mandarin-English acoustic models according to the choice of acoustic units that describe the salient acoustic and phonetic information for Mandarin. To further make use of language information, we jointly train another acoustic model for language identification (LID) with the two acoustic models under the MTL-DNN. To evaluate the effectiveness of our developed MECS-CSR system, extensive experiments are carried out on a public dataset LDC2015S04. It is noted that our approach does not require other language resources. Compared with the first basic MECS-CSR system [1], Mixed Error Rate (MER) of our proposed approach is relatively reduced by 12.49%. The performance improvement benefits from multi-task learning where the common internal representation is obtained from the auxiliary tasks learning.