KNEG-CL: Unveiling data patterns using a k-nearest neighbor evolutionary graph for efficient clustering

Abstract

Existing clustering algorithms often struggle to handle datasets that are diverse and complex, largely due to a dependency on Euclidean distance. Accurately quantifying distances between data points also poses challenges in practical scenarios. Additionally, the curse of dimensionality in high-dimensional datasets also impacts the performance of clustering algorithms. This paper introduces an innovative approach to overcome these challenges: the k-nearest neighbor evolution graph (kNEG), an unweighted directed graph that evolves by incrementally adding directed edges as the value of k increases. This design captures intricate details such as data point density and the direction of density variation. We present kNEG-CL, a clustering algorithm derived from kNEG, which leverages vertex degree and edge directionality to intuitively cluster data. kNEG-CL is guided by two principles: using vertex degrees to identify density peaks, and assessing a balance of outgoing and incoming edges for subcluster merging. By identifying density peaks and utilizing a density-boosting search for initial partitioning, followed by a two-stage merging process, our algorithm achieves high clustering accuracy. Extensive testing across varied datasets demonstrates the superior performance of kNEG-CL, particularly in handling large-scale and high-dimensional data, highlighting its effectiveness in clustering accuracy and computational efficiency.