On the design of an effective corpus for evaluation of Bengali Text Compression Schemes

Abstract

In this paper, we propose an effective platform for evaluation of Bengali text compression schemes. We perform a methodical study on the formulation-approaches of text corpus for data compression and present an effective corpus named Ekushe-Khul for evaluating the Bengali text compression schemes, which is the first initiative in the context of Bengali text compression. To design the Bengali text compression corpus, we consider type to token ratio as the selection criteria with a number of secondary considerations. This paper also presents a mathematical analysis on data compression performance with structural aspects of corpora. The proposed corpus is effective for evaluating compression efficiency of small and middle sized text files.