Research on Real-time Learning Prediction Method Based on Spark

Abstract

Based on the research of real-time prediction and big data processing platform, an effective solution is proposed to solve the shortcomings of current real-time learning prediction in engineering application. By analyzing learners’ learning behaviors related to a certain course, learners’ learning behaviors can be divided into three categories in terms of time and space: online learning behaviors, offline learning behaviors and performance of relevant basic courses. Based on the parallel computation and binary logistic regression algorithm in Spark framework, the off-line learning prediction model is created. In the real-time environment, large scale real-time learning prediction can be realized based on Spark Streaming and kafka. With the increase of learning behaviors data, the scalability problem of prediction scheme can be solved by expanding Spark cluster nodes. The advantages of the proposed scheme have been verified in the practical application of smart campus.