Dynamic monitoring method of mutation event network public opinion based on topic crawler

Abstract

With the rapid development of Internet and the increasing amount of information, it is necessary to use big data technology to solve the bottleneck of processing speed and storage of traditional public opinion monitoring in the era of big data. In this paper, Hadoop open source platform is used to build a big data foundation, realize distributed storage of data, use MapReduce and Spark to realize distributed computing and processing of data, and process the collected data in text. The algorithm model is used to classify and cluster the text information to complete the analysis of text emotional tendency, topic discovery and tracking, and innovatively grasp the public opinion information status of network emergencies. The experimental results from the acquisition rate and average correlation test prove that the algorithm in this paper has higher calculation accuracy. It can provide real-time and effective public opinion analysis service.