Research on Improved Algorithm for Chinese Word Segmentation Based on Markov Chain

Abstract

Chinese words segmentation is an important technique for Chinese web data mining. After the research made on some Chinese word segmentation nowadays, an improved algorithm is proposed in this paper. The algorithm updates dictionary by using Two-way Markov Chain, and does word segmentation by applying an improved Forward Maximum Matching Method based on word frequency statistic. The simulation shows this algorithm can finish word segmentation for a given text quickly and accurately.