Semantic segmentation of camouflage objects via fusing reconstructed multispectral and RGB images

Abstract

Accurate segmentation of camouflage objects in aerial imagery is vital for improving the efficiency of UAV-based reconnaissance and rescue missions. However, camouflage object segmentation is increasingly challenging due to advances in both camouflage materials and biological mimicry. Although multispectral-RGB based technology shows promise, conventional dual-aperture multispectral-RGB imaging systems are constrained by imprecise and time-consuming registration and fusion across different modalities, limiting their performance. Here, we propose the Reconstructed Multispectral-RGB Fusion Network (RMRF-Net), which reconstructs RGB images into multispectral ones, enabling efficient multimodal segmentation using only an RGB camera. Specifically, RMRF-Net employs a divergent-similarity feature correction strategy to minimize reconstruction errors and includes an efficient boundary-aware decoder to enhance object contours. Notably, we establish the first real-world aerial multispectral-RGB semantic segmentation of camouflage objects dataset, including 11 object categories. Experimental results demonstrate that RMRF-Net outperforms existing methods, achieving 17.38 FPS on the NVIDIA Jetson AGX Orin, with only a 0.96% drop in mIoU compared to the RTX 3090, showing its practical applicability in multimodal remote sensing.