Phonetic normalization of microtext

Abstract

Microtext normalization is the challenge of discovering the English words corresponding to the unusually-spelled words used in social-media messages and posts. In this paper, we propose a novel method for doing this by rendering both English and microtext words phonetically based on their spelling, and matching similar ones together. We present our algorithm to learn spelling-to-phonetic probabilities and to efficiently search the English language and match words together. Our results demonstrate that our system correctly handles many types of normalization problems.