Elicit spoken-style data from social media through a style classifier

Abstract

We explore the use of social media data to reduce the effort in developing a conversational speech corpus. The LOTUS-SOC corpus is created by recording Twitter messages through a mobile application. In the first phase, which took around one month, 172 hours of speech from 208 speakers were recorded and ready for use without the need for speech segmentation and transcription. In terms of language similarity to spoken language, the perplexity of LOTUS-SOC with respect to known spoken utterances is lower than that of the broadcast news corpus and almost as low as the telephone conversation corpus. We also applied a style classifier trained from words and parts-of-speech using two machine learning approaches, SVM and CRF, to identify spoken-style utterances in LOTUS-SOC. By training a language model from only the utterances classified as “spoken”, the perplexity of LOTUS-SOC was further reduced as evaluated by three different sets of spoken utterances.