BEST Corpus Development and Analysis

2009 International Conference on Asian Language Processing Pub Date : 2009-12-07 DOI:10.1109/IALP.2009.76

M. Boriboon, Kanyanut Kriengket, P. Chootrakool, Sitthaa Phaholphinyo, Sumonmas Purodakananda, T. Thanakulwarapas, K. Kosawat

引用次数: 12

Abstract

This document describes the development process of the BEST 2009 word segmented-corpus. It is the first corpus to benchmark Thai word segmentation software. The corpus is composed of four genres, namely, collection of news, novels, encyclopedia, and academic articles. It contains 509 files. Its length is 64.1 MB. There are 5,036,229 tokens with 83,027 unique tokens. Common tokens appearing in all genres are 4,556 tokens. They covered 85.13% of the corpus. The highest frequency token in the corpus is ¿¿¿ /thi2/. The first 50 frequency tokens cover 37.65% of the corpus. About 50% of the corpus compose of the first 119 high frequency tokens. All tokens are grouped into 8 categories. Except for Thai spelling category, the other categories play different major parts in specific genres.

查看原文本刊更多论文

BEST语料库开发和分析

本文档描述了BEST 2009分词语料库的开发过程。它是第一个以泰语分词软件为基准的语料库。语料库由四种类型组成，即新闻、小说、百科全书和学术文章。它包含509个文件。其长度为64.1 MB，共有5,036,229个令牌，其中83,027个唯一令牌。在所有类型中出现的常见标记是4,556个标记。它们占语料库的85.13%。语料库中频率最高的标记是¿¿¿/thi2/。前50个频率标记覆盖了语料库的37.65%。大约50%的语料库由前119个高频标记组成。所有的令牌被分成8类。除泰语拼写类别外，其他类别在特定体裁中起着不同的主要作用。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

2009 International Conference on Asian Language Processing

自引率

0.00%

发文量