A Web Crawler-based Consensus Analysis System for Cross-Border Products

Abstract

Internet consensus analysis is an act of analyzing public opinions on the Internet. At present, Internet consensus monitoring and analysis in most enterprises mainly depends on manual work. Manual monitoring of public opinion will encounter many problems: incomplete collection of consensus, untimely discovery of consensus, inaccurate analysis of consensus, and inconvenient use of information. This paper introduces an Internet consensus analysis system and data processing method for cross-border products. This system using web crawler to obtain consensus relating to the products from Internet. In order to turn the raw web pages into structured data, some data extraction and processing methods are used. Also the system can analyze the emotional tendency of the Internet public opinions by emotional dictionary. It has been proved by practice that this system can collect and process the Internet consensus of cross-border products intelligently and automatically.