Information-based massive data retrieval method based on distributed decision tree algorithm

Abstract

Based on the distributed decision tree algorithm, this paper first proposes a method of vertically partitioning datasets and synchronously updating the hash table to establish an information-based mass data retrieval method in a heterogeneous distributed environment, as well as using interval segmentation and interval filtering technologies for improved algorithm of distributed decision tree. The distributed decision tree algorithm uses the attribute histogram data structure to merge the category list into each attribute list, reducing the amount of data that needs to reside in the memory. Second, we adopt the strategy of vertically dividing the dataset and synchronously updating the hash table, select the hash table entries that can be used to update according to the minimum Gini value, modify the corresponding entries and use the hash table to record and control each sub-site. In the case of node splitting, it has a high accuracy rate. In addition, for classification problems that meet monotonic constraints in a distributed environment, this paper will extend the idea of building a monotonic decision tree in a distributed environment, supplementing the distributed decision tree algorithm, adding a modification rule and modifying the generated nonmonotonic decision tree to monotonicity. In order to solve the high load problem of the privacy-protected data stream classification mining algorithm under a single node, a Storm platform for the parallel algorithm PPFDT_P based on the distributed decision tree algorithm is designed and implemented. At the same time, considering that the word vector model improves the deep representation of features and solves the problem of feature high-dimensional sparseness, and the iterative decision tree algorithm GBDT model is more suitable for non-high-dimensional dense features, the iterative decision tree algorithm will be integrated into the word vector model (GBDT) in the data retrieval application, using the distributed representation of words, namely word vectors, to classify short messages on the GBDT model. Experimental results show that the distributed decision tree algorithm has high efficiency, good speed-up and good scalability, so that there is no need to increase the number of datasets at each sub-site at any time. Only a small number of data items are inserted. By splitting some leaf nodes, a small amount is added by branching to achieve a monotonic decision tree. The proposed system achieves a massive data ratio of 54.1% while compared with other networks of massive data ratio.