Unifying RGB and thermal object detection in one detector

Abstract

Object detection in complex real-world environments remains a significant challenge, especially when relying on a single-modality input such as RGB or thermal imagery. This paper proposes Unified Object Detector (UOD), a unified object detection framework that enables accurate RGB, thermal, and RGBT detection using a single set of network parameters. Our key insight reveals that the subtle discriminative features obscured by non-discriminative modality features in either RGB or thermal images can be effectively excavated through cross-modal contrastive learning. Specifically, we design a feature separation framework comprising: 1) Basic Feature Extractor (BFE), 2) Intrinsic Feature Extractor (IFE), and 3) Non-discriminative Modality Feature Extractor (NFE). A cross-modal contrastive learning strategy is proposed between IFE and NFE to decouple modality-invariant intrinsic features from non-discriminative modality features, thus facilitating accurate detection for various kinds of input modalities. Experiments on three benchmarks show that UOD consistently outperforms state-of-the-art single-modality detectors by up to 5% mAP. Furthermore, the architecture-agnostic generalizability of our framework is validated through its successful application to YOLOv5, YOLOv8, and DETR, yielding consistent performance improvements. Code will be released.