RGB-T Object Detection With Failure Scenarios

Abstract

Currently, RGB-thermal (RGB-T) object detection algorithms have demonstrated excellent performance, but issues such as modality failure caused by fog, strong light, sensor damage, and other conditions can significantly impact the detector's performance. This article proposes a multimodal object detection method named diffusion enhanced object detection network (DENet), aiming to address modality failure problems caused by nonroutine environments, sensor anomalies, and other factors, while suppressing redundant information in multimodal data to improve model accuracy. First, we design a multidimensional incremental information generation module based on a diffusion model, which reconstructs the unstable information of RGB-T images through the reverse diffusion process using the original fusion feature map. To further address the issue of redundant information in existing RGB-T object detection models, a redundant information suppression module is introduced, minimizing cross-modal redundant information based on mutual information and contrastive loss. Finally, a kernel similarity-aware illumination module (KSIM) is introduced to dynamically adjust the weighting of RGB and thermal features by incorporating both illumination intensity and the similarity between modalities. KSIM can fine-tune the contribution of each modality during fusion, ensuring a more precise balance that improves recognition performance across diverse conditions. Experimental results on the DroneVehicle and VEDAI datasets show that DENet performs outstandingly in multimodal object detection tasks, effectively improving detection accuracy and reducing the impact of modality failure on performance.