Research on data mining and short-term power load forecasting based on Hadoop

Abstract

With the rapid development of smart grid technology, the amount of power data information increases more and more, and the traditional centralized processing method has been unable to meet the requirements of power system operation. In order to better meet the storage and analysis goals of power data, researchers propose to use distributed power big data. On the basis of understanding the research status of data mining and load prediction algorithms, this paper focuses on the data composite characteristics and change rules of power AI competition according to the big data platform, and constructs the user clustering model with Mahout as the core and the multi-algorithm fusion prediction model with Spark as the core. Combined with the final research results, it is shown that Spark and Mahout, two frameworks in the Hadoop ecosystem, fully consider the advantages and disadvantages of different frameworks, and effectively control the time and cost of experimental analysis. The former can be regarded as a short-term composite forecasting platform, while the latter belongs to a data mining framework.