MBUDet: Misaligned bimodal UAV target detection via target offset label generation

The widespread use of unmanned aerial vehicles (UAVs) has increased the demand for airborne target detection technologies in security and surveillance. The use of only infrared or visible detection technology is often limited by environmental factors and target characteristics. Consequently, the utilization of RGB-Infrared fusion techniques in detection has emerged as a key area of research. However, the alignment operation of multimodal images is quite time-consuming in practical UAV target detection missions. To address this challenge, we propose Misaligned Bimodal UAV Target Detection (MBUDet), which ingeniously integrates the two stages of target alignment and RGB-Infrared object detection into a process, thereby enhancing the detection speed. It primarily comprises four modules: size alignment, target alignment, modal weight calculation, and modal feature fusion. The size alignment module unifies the visible and infrared image sizes; The target alignment module uses existing bimodal target labels to generate target offset labels, which supervise the network to learn target feature alignment, and this module overcomes the effect of mosaic augmentation; the modal weight calculation module mainly solves the problem of a single modality appearing as a target resulting in the network not being able to learn it effectively; the modal feature fusion module focuses on enhancing the feature representations utilizing a spatial attention module. Experiments on our proposed Misaligned Bimodal UAV target dataset (MBU), MBUDet outperforms baseline by 4.8 % and 4.1 % in F1, and AP50 respectively. Also, the experimental results show that the method performs better than existing algorithms. The code associated with this study will be made publicly available soon at the following GitHub repository: http://github.com/Yipzcc/MBUDet.