Reliability Estimators for Classification by Decomposition Method: Experiments in the Medical Domain

Abstract

The performance of a classification system is sometimes unsatisfactory for the needs of real applications. In these cases, the measure of classification reliability should be useful since it takes into account the many issues that influence the achievement of satisfactory results. The most common choice for confidence evaluation consists in using the confusion matrix estimated during the learning phase. As a consequence, the same reliability value is associated with every decision attributing a sample to the same class. In this respect, this paper proposes and compares three different reliability estimators of each classification act of classification systems that belong to the one-per-class framework. They are based on the reliabilities provided by each dichotomizer and are independent of the binary module design. Their performance have been assessed and ranked on private and public medical datasets, showing that one of the estimators outperforms the others.