Classification of text to subject using LDA

Abstract

Blekko Inc., an Internet search company, has divided web sites into subjects we call slash tags. Text from these web sites can be processed using Latent Dirichlet Allocations (LDA), to determine sets of topics for each subject. These topics can then be used to classify any text to determine the subject. We will discuss the methods used to do this; the details of the corpus used for training and testing; and results on how well the system works to classify a priori known text.