Analysis of Hot News Based on Big Data

Abstract

To analyze hot news data of a culture experimental area in China, web crawler, text extraction, named entity recognition, word cloud and other technologies are be used in the paper. The news texts are obtained by using Berkeley DB, Scrapy frame and web page text extraction algorithm firstly. The total number of crawled news articles is 6.87 million. Then based on these data, the analysis of news attention and statistics of hot news to the experimental area are conducted by using NLTK's NER technology and Weka tools. And the relevant industry to the experimental were also analyzed. The visual representation of analysis results also is provided in this paper.