Leveraging frequency and diversity based ensemble selection to consensus clustering

Abstract

Consensus clustering, also called aggregation of clustering (or partitions) is a method that desires to improve the robustness and quality in clustering of a dataset by optimally reconciling the results of different clusterings of the same dataset generated in different ways. This paper proposes a novel way of arriving at a consensus clustering by an ensemble selection strategy. The method avoids considering the entire ensemble and judiciously select few clusterings in the ensemble without compromising on the quality of the consensus. It begins with sorting the ensemble by prioritizing clusterings based on diversity and frequency. It is observed that considering jointly the diversity and frequency helps in identifying few representative partitions that have high potentiality to form qualitatively better consensus than that of the entire ensemble. Finally a greedy strategy is used to select the clusterings in an iterative consensus generation technique that ensures the internal quality of clustering to be monotonically non-decreasing. Empirical results show that the consensus clustering obtained by the proposed algorithm gives better clustering accuracy for many datasets.