Analysis of Big-Data Based Data Mining Engine

Abstract

In order to solve the problem of data mining in big data, this paper studies the data mining engine based on big data. Using Spark as the engine core and programming model, some parallel data mining algorithms are designed and implemented, and an efficient data mining engine system is built. Therefore, the traditional data mining algorithms can run in parallel in the cluster environment, in which big data can be made better of use. Through the above work, a complete big data mining system is realized, which provides an efficient and easy-to-use tool for the implementation of data mining algorithms on big data sets.