Morphological cluster induction of Bantu words using a weighted similarity measure

Abstract

Unsupervised morphological segmentation is attractive for low density languages that have little linguistic description, such as many of the Bantu languages. However, techniques that cluster morphologically related words use string similarity metrics that are more suited for languages that have simple morphological systems. This paper proposes a weighted similarity measure that uses normal distribution for calculating Ordered Weighted Aggregator (OWA) operator weights. The weighting favours shared character sequences that are likely to be part of stems in highly agglutinative languages. The approach is evaluated on text for Chichewa and Citumbuka, both belonging to group N of the Guthrie Bantu languages classification. Cluster analysis results show that the proposed weighted word similarity metric produces better clusters than the Dice Coefficient. Morpheme segmentation results on clusters generated using the OWA weights metric are comparable to the state-of-the-art morphological analysis tools.