Research and application of news-text similarity algorithm based on Chinese word segmentation

Abstract

With the rapid development of the Internet, text messages on the network is also an exponential growth. Facing the vast network of information, how to quickly and efficiently identify the different sites of similar news-text plays a major role in strengthening the integrated management of network information. Existing text similarity algorithm has many disadvantages when used in Chinese news-texts, we propose a more suitable and effective news-text similarity algorithm. This paper uses the Chinese word segmentation technology, and based on this kind of news-text similarity comparison and improved vector space model is applied to the algorithm. Experimental results show that the proposed method is superior to traditional methods the results obtained, thus proving the proposed Chinese news-text similarity calculation method.