Unstructured Text Data Rule Generalization in Attribute-Oriented Induction

Abstract

With the advancement of science and technology, algorithms for data mining are constantly being introduced but most of the algorithms focus on solving the problem of how to reduce the dimension of variables. Nowadays, the amount of data is accumulating and increasing rapidly, and how to compress the amount of data becomes an important topic. Attribute-oriented induction (AOI) is a method of longitudinally compressing data. At present, there are many derivative methods to solve the problem of multi-value-attributes and ordered data. However, few researchers have focused on unstructured text data. Nowadays, the most common way to process text data for classification and induction is to use Latent Dirichlet Allocation (LDA) and generate attributes of the topic. However, the attributes of the topic do not consider whether they exist or not a potential hierarchical relationship. As the topics of text data are hierarchical, we focus on unstructured data such as comment content, transform the comment content into concept datasets, and use AOI to construct a reasonable hierarchical concept. Through induction cost, the final level of induction is determined, too. Finally, the induction rate of the topics generated by the LDA and AOI methods is evaluated to solve the problem of data compression with multiple events and hierarchical relationships among concepts.