基于统计模型的汉语分词比较研究

2010 IEEE International Conference on Software Engineering and Service Sciences Pub Date : 2010-07-16 DOI:10.1109/ICSESS.2010.5552323

Meng Wenchao, Liu Lianchen, Chen Anyan

{"title":"基于统计模型的汉语分词比较研究","authors":"Meng Wenchao, Liu Lianchen, Chen Anyan","doi":"10.1109/ICSESS.2010.5552323","DOIUrl":null,"url":null,"abstract":"Recent years, character based approaches to Chinese word segmentation task are developed, which show great success. In this paper, a detailed comparison among different statistical models are done. Three models (HMM, MEMM and CRF) are considered. First different tag sets are chosen to evaluate the models' precision and efficiency. Then HMM and MEMM are compared with the similar features. At last different features are compared to measure which feature contributes most to Chinese word segmentation. Finally some suggestion is given for developing Chinese word segmentation systems.","PeriodicalId":264630,"journal":{"name":"2010 IEEE International Conference on Software Engineering and Service Sciences","volume":"3 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2010-07-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"4","resultStr":"{\"title\":\"A comparative study on Chinese word segmentation using statistical models\",\"authors\":\"Meng Wenchao, Liu Lianchen, Chen Anyan\",\"doi\":\"10.1109/ICSESS.2010.5552323\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Recent years, character based approaches to Chinese word segmentation task are developed, which show great success. In this paper, a detailed comparison among different statistical models are done. Three models (HMM, MEMM and CRF) are considered. First different tag sets are chosen to evaluate the models' precision and efficiency. Then HMM and MEMM are compared with the similar features. At last different features are compared to measure which feature contributes most to Chinese word segmentation. Finally some suggestion is given for developing Chinese word segmentation systems.\",\"PeriodicalId\":264630,\"journal\":{\"name\":\"2010 IEEE International Conference on Software Engineering and Service Sciences\",\"volume\":\"3 1\",\"pages\":\"0\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2010-07-16\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"4\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"2010 IEEE International Conference on Software Engineering and Service Sciences\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1109/ICSESS.2010.5552323\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"2010 IEEE International Conference on Software Engineering and Service Sciences","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/ICSESS.2010.5552323","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 4

摘要

近年来，基于字符的中文分词方法得到了发展，并取得了很大的成功。本文对不同的统计模型进行了详细的比较。考虑了三种模型(HMM、MEMM和CRF)。首先选择不同的标签集来评估模型的精度和效率。然后比较隐马尔可夫模型和MEMM模型的相似特性。最后对不同的特征进行比较，衡量哪些特征对汉语分词贡献最大。最后对汉语分词系统的发展提出了建议。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

A comparative study on Chinese word segmentation using statistical models

Recent years, character based approaches to Chinese word segmentation task are developed, which show great success. In this paper, a detailed comparison among different statistical models are done. Three models (HMM, MEMM and CRF) are considered. First different tag sets are chosen to evaluate the models' precision and efficiency. Then HMM and MEMM are compared with the similar features. At last different features are compared to measure which feature contributes most to Chinese word segmentation. Finally some suggestion is given for developing Chinese word segmentation systems.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

2010 IEEE International Conference on Software Engineering and Service Sciences

自引率

0.00%

发文量