EGNL-FAT: An Edge-Guided Non-Local network with Frequency-Aware transformer for smoke segmentation

Smoke semantic segmentation presents enormous challenges compared to other objects due to the special characteristics of smoke, such as its non-rigid structure, translucency, and variable modes. Therefore, we design an Edge-Guided Non-Local Network with Frequency-Aware Transformer (EGNL-FAT) for smoke segmentation in real-world scenarios. Unlike traditional smoke segmentation methods, this paper first considers the different feature extraction capabilities of CNNs and transformers due to their varying receptive fields. By constructing a non-local network, the model’s ability to establish long-range dependencies is enhanced. Secondly, edge information is utilized to alleviate the interference caused by intraclass inconsistency and interclass similarity during segmentation. To address the scarcity of real-scene datasets, we introduce the Forest-Scene Smoke Segmentation (FSS) dataset, which contains diverse smoke types and complex backgrounds. We also propose a frequency-aware transformer combining Fast Fourier Transform (FFT) and frequency domain decoupling to reduce computational complexity. A Cross-Domain Fusion Module (CDFM) is introduced for collaborative learning of information from various sources using weighted fusion and Coordinate Attention (CA). Additionally, we develop an Edge Feature Extraction Module (EFEM) that efficiently extracts detailed edge information at full resolution using a shallow structure. Finally, a Multi-Directional Cross Attention (MDCA) mechanism is proposed to compute similarities between edge and decoder feature maps, guiding accurate segmentation. Experimental results show that EGNL-FAT achieves mean Intersection over Unions (mIoUs) of 59.66 % (FS01) and 64.68 % (FS02) on FSS, and 79.04 % on SMOKE5K. It demonstrates excellent performance while maintaining satisfactory model complexity and processing efficiency. Our code and dataset are publicly available at: https://github.com/yitccc/smoke-segmentation/tree/master.