Exploring the parameter state space of stacking

Abstract

Ensemble learning schemes are a new field in data mining. While current research concentrates mainly on improving the performance of single learning algorithms, an alternative is to combine learners with different biases. Stacking is the best-known such scheme which tries to combine learners' predictions or confidences via another learning algorithm. However, the adoption of stacking into the data mining community is hampered by its large parameter space, consisting mainly of other learning algorithms: (1) the set of learning algorithms to combine, (2) the meta-learner responsible for the combining; and (3) the type of meta-data to use - confidences or predictions. None of these parameters are obvious choices. Furthermore, little is known about the relation between the parameter settings and performance of stacking. By exploring all of stacking's parameter settings and their interdependencies, we attempt to make stacking a suitable choice for mainstream data mining applications.