Selective output smoothing regularization: Regularize neural networks by softening output distributions

Abstract

Convolutional neural networks (CNNs) often exhibit overfitting due to overconfident predictions, which limits the effective utilization of training samples. Inspired by the diverse effects of training from different samples, we propose selective output smoothing regularization(SOSR) that improves model performance by encouraging the generation of equal logits on incorrect classes when handling samples that are correctly and overconfidently classified. This plug-and-play approach integrates seamlessly into diverse CNN architectures without altering their core design. SOSR demonstrates consistent improvements on various benchmarks, such as a 1.1% accuracy gain on ImageNet with ResNet-50 (77.30%). It synergizes effectively with several widely used techniques, such as CutMix and label smoothing, achieving incremental benefits, highlighting its potential as a foundational tool in advancing deep learning applications. Overall, SOSR effectively alleviates underutilization of high-confidence samples, enhances the generalizability of CNNs, and emerges as a robust tool for improving deep learning applications.