Reshaping the Semantic Logits for Proposal-free Panoptic Segmentation

Abstract

We propose to enable the general semantic segmentation frameworks to separate instances so that such frameworks can be used for the panoptic segmentation task. In the semantic segmentation frameworks, the logits which are output from the neural network and normalized by the following softmax function can only distinguish classes but not instances. In this work, we find simple regularization on the logits can help to single out the instances, which is modeled by an energy-based representation, energy surface. Several regularization approaches are discussed and a novel persistent homology-based instance extraction method is proposed to obtain the instances. Finally, we demonstrate the generality of the logit regularization on different base semantic segmentation frameworks and evaluating them on Cityscapes, Mapillary Vistas, and COCO. High-quality semantic segmentation frameworks such as DeepLabV3+ and HRNet-OCR can achieve competitive performance to the state-of-the-art proposal-free panoptic segmentation solver. Codes and trained models will be made public.