AdaBoost.SDM: Similarity and dissimilarity-based manifold regularized adaptive boosting algorithm

Abstract

AdaBoost is a successful ensemble learning algorithm that generates a sequence of base learners, where each base learner is encouraged to focus more on those data points that are misclassified by the previous learner. That being said, AdaBoost, in its original form, lacks any mechanism to explicitly leverage the underlying geometric structure of data or manifold. Recent studies have shown that a training process that penalizes model outputs that do not align with the data manifold can lead to better generalization. In this paper, we aim to define a convex objective function for training AdaBoost that enforces a smooth variation of the model predictions over the data manifold. In this regard, we adopt a mixed-graph Laplacian that in contrast with the conventional Laplacian regularization can handle both label similarity and dissimilarity knowledge between data points. Compared with the original form of AdaBoost, the results demonstrate the effectiveness of the proposed similarity and dissimilarity-based manifold regularized AdaBoost (AdaBoost.SDM) in exploiting the data manifold and, at the same time, encoding the label similarity and dissimilarity to improve the classification performance. Our experimental results show that AdaBoost.SDM is highly competitive with state-of-the-art manifold regularized algorithms, including LapRLS and LapSVM.