Impact of Inter-Modality Misalignment on Multispectral Object Detection

Abstract

Object Detection significantly benefits from combining complementary spectral modalities; however, its performance suffers greatly from misregistration between sensor streams due to spatial misalignments, viewpoint differences between sensors, and inconsistent annotations, a common feature of multi-modal datasets used in object detection system development. In this work, we address the critical challenge of modality misregistration. We propose a comprehensive alignment framework based on object-aware local homography estimation, anchored on Long-Wave Infrared Radiation (LWIR) annotations, to generate a spatially aligned training dataset for improved model development. For our experiments, we utilise the Multispectral Object Detection (MOD) dataset, which includes RGB, Short-Wave Infrared Radiation (SWIR), Mid-Wave Infrared Radiation (MWIR), and Long-Wave Infrared Radiation (LWIR) imagery. The proposed alignment pipeline ensures a consistent field of view, resolution normalization, and accurate cross-modality annotation projection. To quantify the impact of alignment, we benchmark four state-of-the-art fusion architectures (DEYOLO, TarDAL, CMTFusion, and U2Fusion) on both the original and the aligned versions of the MOD dataset. Experimental results demonstrate significant improvements of 10 to 30 percentage points in mAP@0.5 after training on the aligned MOD data, with enhanced localization accuracy and reduced false detections. We further verified the generalizability of the proposed alignment framework on a different dataset (FLIR ADAS), observing 3.54 percentage points improvement in mAP@0.5 after training on the aligned RGB-LWIR image pairs. U2Fusion emerged as the architecture most tolerant to misregistration in initial experiments, and this architecture is used to conduct a further sensitivity analysis to evaluate model robustness against real-world misalignments, in a controlled way. The results of this sensitivity analysis could be useful for designers of practical systems, where small misalignment errors between image modalities commonly occur.