On Validation Setup for Multiclass Imbalanced Data Sets

Abstract

The validation of experiments is commonly evaluated with Cross-Validation methods. In the literature the 10-fold, followed by bootstrap, are the most indicated methods. However there lacks a study of a proper validation procedure for imbalanced data sets, specially for the rare class case. In this work the most used validation methods were tested in ten imbalanced data sets, with a generic and an ad hoc classifiers. Analyses showed that 10-fold, followed by hold-out, are the indicated methods when using a generic classifier. For the ad hoc classifier the 10-fold, followed by bootstrap, are the indicated ones. In the case of rare classes in a data set, the most indicated method is the repeated hold-out.