Ensemble Learning Through Evolutionary Multitasking: A Formulation and Case Study

Abstract

Evolutionary machine learning has drawn much attentions on solving data-driven learning problem in the past decades, where classification is a major branch of data-driven learning problem. To improve the quality of obtained classifier, ensemble is a simple yet powerful strategy. However, gathering classifiers for ensemble requires multiple runs of learning process which bring additional cost at evaluation on the data. This study proposes an innovative framework for ensemble learning through evolutionary multitasking, i.e., the evolutionary multitasking for ensemble learning (EMTEL). There are four main features in the EMTEL. First, the EMTEL formulates a classification problem as a dynamic multitask optimization problem. Second, the EMTEL utilizes evolutionary multitasking to resolve the dynamic multitask optimization problem for better convergence through the synergy of common properties hidden in the tasks. Third, the EMTEL incorporates evolutionary instance selection for saving the cost at evaluation. Finally, the EMTEL formulates the ensemble learning problem as a numerical optimization problem and proposes an online ensemble aggregation approach to simultaneously select appropriate ensemble candidates from learning history and optimize ensemble weights for aggregating predictions. A case study is investigated by integrating two state-of-the-art methods for evolutionary multitasking and evolutionary instance selection respectively, i.e., the symbiosis in biocoenosis optimization and cooperative evolutionary learning and instance selection. For online ensemble aggregation, this study adopts the well-known covariance matrix adaptation evolution strategy. Experiments validate the effectiveness of the EMTEL over conventional and advanced evolutionary machine learning algorithms, including genetic programming, self-learning gene expression programming, and multi-dimensional genetic programming. Experimental results show that the proposed framework ameliorates state-of-the-art methods, and the improvements on quality for multiclass classification are at 8.48% at least and 56.35% at most in relation to the macro F-score. For convergence speed, the speedups achieved by the proposed framework are 7.85 at least and 100.53 at most on multiclass classification.