Feature Abstraction for Lightweight and Accurate Chinese Word Segmentation

Abstract

Chinese word segmentation (CWS) is an important and necessary problem to analyze Chinese texts. The state-of-art CWS systems are mostly based on sequence labeling algorithm and use the discriminative model with millions of overlapping binary features. However, there are few works on porting these systems to the devices with limited computing capacity and memory. In this paper, we focus on two challenges in Chinese word segmentation: (1) low accuracy of out-of-vocabulary word and (2) huge feature space. To resolve these two difficult problems, we propose a method to abstract the original input on both character and feature levels. We group the "similar'' features to generate more abstract representation. Experimental results show that feature abstraction can greatly reduce the feature space with a comparable performance.