Small Sample Estimation of Classification Metrics

Abstract

Estimation of classification evaluation metrics for small sample classification data is one of the most challenging task in Machine Learning model building process. The value of an evaluation metric based on the test data is considered to be the generalized performance measure of a classification model. Measuring the performance of a model based only on the test data works well when we have a large data set and both train and test data follow the same population distribution. However, this assumption does not always work for small data set. Therefore, measuring performance of a model based on test data set may mislead us for small sample classification problems. To deal with such situations, in this paper, we propose a modified method to estimate classification evaluation metrics. The new method provides an improved and consistent estimation of classification evaluation metrics.