MST: A Modified Sparse Transformer with depth-aware attention for multi-modal camera–LiDAR fusion in autonomous vehicles

Abstract

Sensor fusion plays a pivotal role in enhancing the accuracy, safety, and decision-making capabilities of autonomous vehicles by integrating camera and LiDAR data. Cameras provide rich semantic information, while LiDAR offers precise depth estimation; their fusion is crucial for robust perception in complex driving scenarios. Transformer-based models have emerged as effective tools for multimodal fusion by leveraging self-attention to capture intricate relationships between sensor data. However, traditional transformers face computational efficiency challenges with long input sequences and sparse data. To address these limitations, we propose the Modified Sparse Transformer (MST) for camera–LiDAR fusion. The MST reduces attention matrix complexity, enabling faster processing while maintaining high performance with fewer parameters. Key innovations include depth-aware attention mechanisms, cross-modal feature alignment, and dynamic instance interaction modules. These collectively enhance object detection accuracy in challenging conditions such as low visibility and dense traffic. Experiments on benchmark datasets demonstrate significant improvements in both accuracy and efficiency compared to existing methods.