HybridFusion: A Universal Multimodal Data Fusion Framework for 3-D Detection and Tracking

Abstract

Environment perception based on multimodal data fusion is crucial for the development of autonomous driving (AD) technology. However, the existing fusion-based detection and tracking methods often struggle to efficiently utilize the multimodal complementary information. This limitation can significantly impact the safety of autonomous vehicles (AVs), especially in complex traffic scenarios and adverse weather conditions. To address this issue, this work proposes a universal multimodal data fusion framework called HybridFusion for 3-D object detection and tracking tasks. Specifically, a novel Modal-Specific Feature Interaction Strategy (MSFIS) is introduced within the detection algorithm. The strategy complementarily fuses multimodal features while preserving the potential information from individual modalities. Furthermore, a Motion-Appearance Coupling Driven Data Association Strategy (MACD-DAS) is incorporated into the tracking algorithm. The strategy comprehensively processes spatial motion information and multimodal appearance features, effectively reducing the uncertainty in state prediction and feature matching. Extensive experiments were conducted on View-of-Delft (VoD) and K-Radar datasets. The experimental results demonstrate that the proposed HybridFusion significantly improves the performance of multimodal detection and tracking algorithms in complex traffic scenes and adverse weather conditions. The demo is available at https://youtu.be/el_5jtxDO2I