EntroFormer: An entropy-based sparse vision transformer for real-time semantic segmentation

Abstract

Image semantic segmentation plays a fundamental role in a wide range of pixel-level scene understanding tasks. State-of-the-art segmentation methods often leverage sparse attention mechanisms to identify informative patches for modeling long-range dependencies, significantly reducing the computational complexity of Vision Transformers. Most of these methods focus on selecting regions that are highly relevant to the queries, achieving strong performance in tasks like classification and object detection. However, in the semantic segmentation task, current sparse attention methods are limited by their query-based focus, overlooking the importance of interactions between different objects. In this paper, we propose Sparse Entropy Attention (SEA) to select regions with higher informational content for long-range dependency capture. Specifically, the information entropy of each region is computed to assess its uncertainty in semantic prediction. Regions with high information entropy are considered informative and selected to explore sparse global semantic dependencies. Based on SEA, we present an entropy-based sparse Vision Transformer (EntroFormer) network for real-time semantic segmentation. EntroFormer integrates sparse global semantic features with dense local ones, enhancing the network’s ability to capture both the interaction of image contents and specific semantics. Experimental results show that the proposed real-time network outperforms state-of-the-art methods with similar parameters and computational costs on the Cityscapes, COCO-Stuff, and Bdd100K datasets. Ablation studies further demonstrate that SEA outperforms other sparse attention mechanisms in semantic segmentation.