Vision transformer enhanced with convolutional attention and graph convolution for semantic segmentation

Abstract

Semantic segmentation is a dense prediction task that assigns semantic labels to every pixel in an image. Effectively modeling global contextual information is a primary challenge in this task. Recently, some methods using Vision Transformer (ViT) encoders based on self-attention mechanisms have shown significant performance improvements. However, encoding spatial information purely through self-attention mechanisms tends to provide a more holistic representation and performs inadequately in handling object details. To address this, we propose a stripe depth-wise convolutional attention (SDCA) module. This module aggregates local convolution features at multiple scales as its attention map. Utilizing attention map generated by convolution at different scales effectively compensates for the limitations of self-attention mechanisms in handling object details. Additionally, to ensure the generation of more coherent predictions, we introduce a spatial feature graph convolution (SFGC) module to explicitly model the spatial relationships between patches. We apply these two modules in parallel to the output features of the Transformer block and add their output features to the original features for subsequent layer learning. Our method achieved mIoU scores of 50.5%, 59.1% and 55.0% on the COCO-Stuff-10K, PASCAL-Context and ADE20K datasets, respectively, surpassing some of the recent state-of-the-art methods.