Network Fission Ensembles for low-cost self-ensembles

Abstract

Recent ensemble learning methods for image classification have demonstrated improved accuracy with low extra cost. However, they still rely on multiple trained models for ensemble inference, which can become a significant burden as the model size grows. Moreover, their performance has been somewhat limited compared to Deep Ensembles, primarily due to the lower performance of individual ensemble members. In this paper, we propose a low-cost ensemble learning and inference method called Network Fission Ensembles (NFE), which transforms a conventional network into a multi-exit structure allowing predictions to be made at different stages and enabling ensemble learning. To achieve this, we group the weight parameters in the layers into several sets and create multiple auxiliary paths by combining each set to construct multi-exits. We call this process Network Fission. Since this process simply changes the existing network structure to have multiple exits (i.e., classification outputs) without using additional networks, there is no extra computational burden. Furthermore, we employ an ensemble knowledge distillation technique exploiting the losses of all exits to train the network, so that we can achieve high generalization performance despite the reduced network size of each path composed of pruned weights. With our simple yet effective method, we achieve an accuracy of 83.5% on CIFAR100 with Wide-ResNet28-10, surpassing the best existing ensemble method, Deep Ensembles, which achieves 83.0%, while only one-third of the computational complexity is required in our method. The code is available at https://github.com/hjdw2/NFE.