Research on Parallel Data Mining Based on Spark

Abstract

In the current era of big data, the rapid development of network technology and hardware equipment leads to exponential data growth. However, under the challenge of massive data, there are still some problems in the field of data mining, such as low efficiency of algorithm execution, insufficient parallel optimization of algorithms and poor usability of data mining platforms. This paper focuses on parallel data mining algorithms and parallel data mining tools. Based on Spark as a programming model and processing engine, a distributed parallel data mining scheduling framework is designed and implemented based on Hadoop and Spark, which can meet the needs of users for mining and analyzing large data sets. The scheduling system implements common data mining algorithms such as classification, prediction, clustering and data preprocessing, and can complete data mining modeling by visual drag and drop algorithm program.