Standardization of expected value in gap statistic using Gaussian distribution for optimal number of clusters selection in K-means

Abstract

K-means clustering is a widely used unsupervised learning technique for partitioning data into distinct groups. However, determining the Optimal Number of Clusters (ONC) remains a significant challenge due to the subjective nature of existing methods. The Gap Statistic is a common approach for ONC selection, yet it has limitations across various data scenarios. To address these challenges, this paper introduces the Enhanced Gap Statistic (EGS), which improves the traditional Gap Statistic by incorporating a Gaussian distribution to standardize reference data and integrating an adjustment factor to enhance ONC selection accuracy. In this study, we apply the Gaussian distribution to generate the reference dataset in the Gap Statistic due to its stability, efficiency, and robustness in handling outliers. While Gaussian assumptions work well in many cases, we acknowledge that they may not always be suitable, particularly for skewed, heavy-tailed, or multimodal data. In such scenarios, alternative approaches, such as t-distribution and kernel density estimation, may provide better adaptability. Furthermore, we recognize that the computational complexity of incorporating Gaussian standardization could impact scalability for large datasets, necessitating further optimizations. To evaluate EGS, we compare its performance against widely used clustering validation metrics, including the Davies-Bouldin index, Calinski-Harabasz index, Silhouette index, Elbow curve, and the conventional Gap Statistic. Experimental results demonstrate that EGS consistently outperforms traditional methods in both accuracy and computational efficiency. Specifically, EGS achieved efficiency values of 0.0500, 0.21, 0.12, 3.340 and 4.34 s and accuracy values of 89.35 %, 95.35 %, 80.35 %, 74.3 % and 97.3 for Time Series, Well Log, Hitter, the large-scale Traffic Crash dataset and the high dimensional Darwin Dataset, respectively. The findings shows that EGS as a highly effective and computationally efficient method for ONC selection, making it a valuable tool for complex and large-scale data environments.