Internet Newborn Word Recognition Method under Conditional Random Field Model

This paper proposes an approach of automatic detection of new words. It analyzes the webpages acquired from Internet on a large scale and detect new words. According to morphological rules it will perform further filtering on detection results to extract existed new words in the corpus. Our scheme adopts conditional random field and puts forward twp improvement: in the new word detection stage, a high efficient left (right) entropy calculation method is proposed to improve the detection speed, which effectively reduces the influence of unrelated characters in the calculation; then, a quantized model of missing logged words is also proposed, which is based on participle to extract repeated strings, and it can be used to evaluate the problem of missing words. The CRF based combination model proposed in this paper is proved to be a very effective new word detection method, whether from the open experiment effect or the generalization ability of the model.