Empirical Study of Chinese Text Similarity Computation Based on Machine Translation

Abstract

For the problems of Chinese text similarity calculation based on word frequency statistics, this paper proposed a method by using machine translation to translate Chinese text into English text, indirectly calculate similarity of given texts. This method can avoid some shortcomings of Chinese word segmentation and utilize the advantages of the natural word segmentation of English, and also can use machine translation to indirectly take the semantics of part of words into account. The experiments compared it with the way of directly using Chinese, and a detailed analysis was performed. Experiments show that this method can improve most of social texts' similarity computation as well as increase the accuracy of the computation as a whole.