Deep negative correlation classification

Abstract

Ensemble learning serves as a straightforward way to improve the performance of almost any machine learning algorithm. Existing deep ensemble methods usually naïvely train many different models and then aggregate their predictions. This is not optimal in our view from two aspects: (1) Naïvely training multiple models adds much more computational burden, especially in the deep learning era; (2) Purely optimizing each base model without considering their interactions limits the diversity of ensemble and performance gains. We tackle these issues by proposing deep negative correlation classification (DNCC), in which the accuracy and diversity trade-off is systematically controlled by decomposing the loss function seamlessly into individual accuracy and the “correlation” between individual models and the ensemble. DNCC yields a deep classification ensemble where the individual estimator is both accurate and “negatively correlated”. Thanks to the optimized diversities, DNCC works well even when utilizing a shared network backbone, which significantly improves its efficiency when compared with most existing ensemble systems, as illustrated in Fig. 2. Extensive experiments on multiple benchmark datasets and network structures demonstrate the superiority of the proposed method.