Interpretable categorical data clustering via hypothesis testing

Abstract

Categorical data clustering algorithms are extensively investigated but it is still challenging to explain or understand their output clusters. Hence, it is highly demanded to develop interpretable clustering algorithms that are capable of explaining categorical clusters in terms of decision trees or rules. However, most existing interpretable clustering algorithms focus on numeric data and the development of corresponding algorithms for categorical data is still in the infant stage. In this paper, we tackle the problem of interpretable categorical data clustering by growing a binary decision tree in an unsupervised manner. We formulate the candidate split evaluation issue as a multiple hypothesis testing problem, where the null hypothesis posits that there is no association between each attribute and the candidate split. Subsequently, the $p$-value for each candidate split is calculated by aggregating individual test statistics from all attributes. Thereafter, a significance-based splitting criteria is established. This involves choosing an optimal split with the smallest $p$-value for tree growth and using a significance level to stop the non-significant split. Extensive experimental results on real-world data sets demonstrate that our algorithm achieves comparable performance in terms of cluster quality and explainability relative to those of state-of-the-art counterparts.