Application of Conditional Random Fields model in unknown words identification

Abstract

This paper proposed a method for Unknown Words Identification (UWI) based on repeats. To identify Unknown words with reliable theory, we put forward a formal model for the process of UWI, which can give directions on the selection of features used in UWI in theory. For the formal model, we propose employing Conditional Random Fields model (CRF) as statistical frame to resolve it. Under the statistical frame, UWI is converted to the process of exploiting effective features that can represent the essences of unknown words. The experiments show that the method of this paper is effective, and reasonable combination of features used in CRF can evidently improve the result of UWI. The ultimate result (F score) of this method is 47.81% and 69.83% in open test and word extraction respectively, which is better over the best result reported in previous works.