HFFN: Hierarchical feature fusion network for RGB-infrared object detection

Abstract

In recent years, multimodal object detection techniques based on visible(RGB) and infrared(IR) images have gained attention as a research hotspot. These techniques leverage the complementary characteristics of RGB and IR images to overcome the environmental adaptability limitations of single-modality methods. The fusion of RGB and IR images is crucial for object detection under complex lighting conditions. By effectively integrating the complementary information of the two modality images, the robustness and accuracy of detection are enhanced. However, current image fusion methods often encounter issues of inaccurate modality alignment and insufficient representation of fused features during feature extraction and fusion. We propose a Hierarchical Feature Fusion Network (HFFN) that achieves efficient extraction and accurate alignment of multimodal features using an Adaptive Pooling Transformer (APT) and an Invertible Detail Extraction (IDE), significantly enhancing the fusion quality. The APT combines pooling operations and the Transformer structure, enabling it to capture global information and expand the receptive field of features; The IDE focuses on extracting local detail features, including edges and texture information, to enhance the fineness and clarity of the fused image. We constructed object detectors based on the HFFN to validate its effectiveness. Extensive experiments on datasets demonstrate the superior performance of our method.