Research of PM2.5 Real-Time Prediction Model in Spark Cluster Environment

Abstract

Big data technologies provide new ideas and means for statistical prediction of environmental air quality. In this paper, how to construct PM2.5 real-time prediction model for monitoring stations by using R language in Spark clusters is studied. Real-time data of monitoring stations stored in traditional relational database are converted into target dataset which can be put into cluster and processed by R language. The correlation analysis of pollutants and meteorological parameters that affecting the PM2.5 is carried out, so that to determine the input variables of multiple linear regression for constructing PM2.5 real-time prediction model. In spark cluster environment, Sparklyr and MLib packages are used by R language to construct prediction models for monitoring stations, each model is evaluated by four aspects such as residual analysis, significance detection, decision coefficient and test set prediction to justify its effectiveness. The experiment result shows that the model can be used to predict PM2.5 real-time value accurately.