Semi-Supervised Semantic Segmentation of Class-Imbalanced Images: A Hierarchical Self-Attention Generative Adversarial Network

Abstract

How to train models with unlabeled data and implement one trained model across several data sets are key problems in computer vision applications that require high-cost annotations. Recently, a generative model [1] proves its advantages in semi-supervised segmentation and out-of-domain generalization. However, this method becomes less effective when meet with class-imbalanced images whose foreground occupies small areas. To solve this problem, we introduce a hierarchical generative model with a self-attention mechanism to help with capturing features of foreground objects. Concretely, we apply a two-stage hierarchical generative model to perform image synthesis with the self-attention mechanism. Since attention maps are also semantic labels in segmentation fields, the hierarchical self-attention model can synthesize images and corresponding segmentation labels simultaneously. At test time, the segmentation is achieved by mapping input images into latent presentations with two encoders and synthesizing labels with the generative model. We evaluate our hierarchical model on three biomedical segmentation data sets. The experimental results demonstrate that our method outperforms other baselines on semi-supervised segmentation of class-imbalanced images, and meanwhile, pre-serves out-of-domain generalization ability.