Ensemble classifiers with improved overfitting

Overfitting has been always considered as a challenging problem in designing and training of ensemble classifiers. Obviously, the use of complex multiple classifiers may increase the success of ensemble classifier in feature space division with intertwined data and also may decrease the training error to minimum value. However, this success does not exist on the test data. Ensemble classifiers are more prone to overfitting than single classifiers because ensemble classifiers have been formed of several base classifiers and overfitting occurrence in each base classifier can transfer the problem to the final decision of the ensemble. In this paper, after quantitative and qualitative analysis of overfitting, a solution for improving overfitting is proposed by using heuristic algorithms. In this way, Multi-Objective Inclined Planes Optimization (MOIPO) and Multi-Objective Particle Swarm Optimization (MOPSO) are used and their results are compared with each other. Simulation results show that the simultaneous minimization of ensemble size and error rate in the training phase, can lead to a significant reduction in the amount of overfitting. In fact, with this approach in the training phase, the ensemble classifier is required to minimize the error with the most simple and minimum number of base classifiers and therefore overfitting is prevented. However, previous researches related to overfitting have ignored the ensemble size as an objective function.