Enhancing 3D Visual Grounding with Deformable Attention Transformer and Geometry Affine Transformation: Overcoming sparsity challenges

Abstract

In this paper, we introduce 3DVG-Deformable-Attention Transformer (3DVG-DT), a novel framework designed to address the challenge of imprecise target object localization in 3D Visual Grounding (3DVG) due to point cloud sparsity. By integrating Deformable Attention Transformer (DAT) and Geometry Affine Transformation (GAT), 3DVG-DT effectively mitigates the effects of point cloud sparsity and irregularity, significantly improving 3DVG accuracy. We propose a Dual-Mode Feature Fusion (DMF) module for object detection and matching within complex point clouds, while a Description-aware Keypoint Affine Transformation Sampling (DKAS) strategy further enhances performance. Leveraging DeBERTa-V3 for language encoding, we demonstrate the effectiveness of 3DVG-DT on ScanRefer and Referit3D datasets, showcasing improved target detection capabilities under sparse point cloud conditions. Experimental results reveal substantial gains over existing methods, particularly in handling sparse point clouds.