Increase the sensitivity of moderate examples for semantic image segmentation

Dominant paradigms in modern semantic segmentation resort to the scheme of pixel-wise classification and do supervised training with the standard cross-entropy loss (CE). Although CE is intuitively straightforward and suitable for this task, it only cares about the predicted score of the target category and ignores the probability distribution information. We further notice that fitting hard examples, even if their number is small, results in model over-fitting in the test stage, as accumulated CE losses overwhelm the model during training. Besides, a large number of easy examples may also dazzle the model training. Based on this observation, this work presents a novel loss function we call Sensitive Loss (SL), which utilizes the overall predicted probability distribution information to down-weight the contribution of extremely hard examples (outliers) and easy examples (inliers) during training and rapidly focuses model learning on moderate examples. In this manner, SL encourages the model to learn potential feature generalities rather than diving into the details and noise implied by outliers to the extent. Thus, it is capable of alleviating over-fitting and improving generalization capacity. We also propose a dynamic Learning Rate Scaling (LRS) strategy to alleviate the decreasing gradient and improve the performance of SL. Extensive experiments evidence that our Sensitive Loss is superior to existing handcrafted loss functions and on par with searched losses, which generalize well to a wide range of datasets and algorithms. Specifically, training with the proposed SL brings a notable 1.7% mIoU improvement for the Mask2Former framework on Cityscapes dataset off the shelf.