Experiments on simultaneous combination rule training and ensemble pruning algorithm

Abstract

Nowadays many researches related to classifier design are trying to exploit strength of the ensemble learning. Such hybrid approach looks for the valuable combination of individual classifiers' outputs, which should at least outperforms quality of the each available individuals. Therefore the classifier ensembles are recently the focus of intense research. Basically, it faces with two main problems. On the one hand we look for the valuable, highly diverse pool of individual classifiers, i.e., they are expected to be mutually complimentary. On the other hand we try to propose an optimal combination of the individuals' outputs. Usually, mentioned above tasks are considering independently, i.e., there are several approaches which focus on the ensemble pruning only for a given combination rule, while the others works are devoted to the problem how to find an optimal combination rule for a fixed line-up of classifier pool. In this work we propose to put ensemble pruning and combination rule training together and consider them as the one optimization task. We employ a canonical genetic algorithm to find the best ensemble line-up and in the same time the best set-up of the combination rule parameters. The proposed concept (called CRUMP - simultaneous Combination RUle training and enseMble Pruning) was evaluated on the basis the wide range of computer experiments, which confirmed that this is the very promising direction which is able to outperform the traditional approaches focused on either the ensemble pruning or combination rule.