Evaluation and Verification of NLP Datasets for the Albanian Language

Abstract

Computational Linguistics has seen tremendous growth and provided users with high end applications in the form of automatic translation tools, speech recognition, speech synthesis etc. However, such advancements are lacking for low resource languages. Our research aims to tackle one of these challenges, specifically advancing Computational Linguistics and Natural Language Processing for the Albanian Language. To develop accurate NLP tools, one must have a consistent and clean dataset for that language. In this paper we evaluate two well-known text corpora: OSCAR and CCAligned. The results are compared with a dataset that we have collected and curated, which we will refer in this paper as alb_dataset. Various statistical means have been used to compare and evaluate the datasets. Conclusions of this paper can be used by NLP researchers of the Albanian language before they use one of the text corpora mentioned above.