PMDFN3D: Pre-mid dual fusion network for 3D object detection

Abstract

In recent years, multi-modality 3D object detection technology is gradually becoming the mainstream of 3D object detection. In multi-modality 3D object detection, effectively fusing information from point cloud data and image data remains a significant challenge. Existing multi-modality 3D object detection models mainly use one of the pre-, mid- or post-fusion strategies to fuse image data and point cloud data, and each of these fusion strategies has some shortcomings. Currently, integrating multiple fusion strategies into a framework is still a research gap in the field of multi-modality 3D object detection. To fill this gap, we propose a pre-mid dual fusion network for 3D object detection (PMDFN3D), which skillfully integrates the pre-fusion and mid-fusion into a unified framework. Specifically, we first design a depth-guided cross-modality feature fusion module that enables the effective integration of image and point features without requiring complex feature alignment operations. Then, we design a neighboring feature interaction attention module to mitigate the impact of down-sampling operations in the point cloud backbone network on the precision of point features. Finally, we design a simple object-level feature selector and an object-level feature-guided cross-modality feature fusion module, which adaptively integrate image features relevant to the objects with object-level point features. Experimental results on the SUN RGB-D dataset demonstrate that our network has achieved state-of-the-art performance in 3D object detection.