Bootstrapping vision–language transformer for monocular 3D visual grounding

In the task of 3D visual grounding using monocular RGB images, it is a challenging problem to perceive visual features and accurately predict the localization of 3D objects based on given geometric and appearances descriptions. Traditional text-guided attention-based methods have achieved better results than baselines, but it is argued that there is still potential for improvement in the area of multi-modal fusion. Thus, Mono3DVG-TRv2, an end-to-end transformer-based architecture that employs a visual-text multi-modal encoder for the alignment and fusion of multi-modal features, incorporating an enhanced transformer module proven in 2D detection, is introduced. The depth features predicted by the multi-modal features and the visual-text features are associated with the learnable queries in the decoder, facilitating more efficient and effective acquisition of geometric information in intricate scenes. Following a comprehensive comparison and ablation study on the Mono3DRefer dataset, this method achieves state-of-the-art performance, markedly surpassing the prior approach. The code will be released at https://github.com/Jade-Ray/Mono3DVGv2.