Real-time Semantic Segmentation with Bilateral Patch Attention

Semantic segmentation, a fundamental task in computer vision, has evolved significantly with the introduction of deep learning techniques, particularly fully convolutional networks (FCNs). In the context of real-time semantic segmentation, the demand for efficient yet accurate models has grown, particularly for resource-constrained devices. Recent advancements have explored the fusion of global context and local details through bidirectional network structures, exemplified by BiSeNet, STDCNet, and DDRNet. However, issues of pixel inconsistency within the same object classification persist. Attention-based models like SETR and SegFormer have shown promise in mitigating this issue by capturing intricate spatial dependencies. This paper introduces the concept of ‘Mask Disarrange’ and proposes a lightweight attention mechanism suitable for real-time semantic segmentation. The Cross Patch Attention (CPA) and Inter Patch Attention (IPA) methods are presented, addressing fusion and mask disarrange challenges while maintaining computational efficiency. Experimental results on the Cityscapes dataset demonstrate the effectiveness of the proposed Bilateral Patch-Net (BPNet) in achieving superior segmentation performance and increased frames per second (FPS) compared to the state-of-the-art PIDNet. BPNet's contributions lie in its simplicity, efficiency, and applicability to diverse domains, offering potential for broader adoption in computer vision applications.