The analysis on mistaken segmentation of Tibetan words based on statistical method

Abstract

In this paper, by using the Tibetan word segmentation system, IEA-TWordSeg, the authors attempt segmentation of the total 1271 sentences in the closed set and 1000 sentences in an open set. The accuracy of testing is 99.54% and 92.41% respectively. The authors describe the wrong segmentation types as well as the causes of the mistakes, and demonstrate the proportion of different types of segmentation errors. The purpose of the article is to provide clues for those who intend to improve the accuracy of Tibetan word segmentation system.