CrysFormer++: Dual-phase refinement learning for transparent object depth estimation

Abstract

Transparent object depth estimation is a critical yet challenging task in robotic perception, particularly in grasping applications for industrial automation and human-robot interaction. Due to the high transmittance of visible light in transparent materials, depth sensors often suffer from severe depth measurement errors, leading to inaccuracies in grasp planning and object manipulation. To address this issue, we propose a Mamba-Transformer hybrid encoding framework (CrysFormer++) for robust depth estimation of transparent objects. The model integrates VMamba to efficiently model global long-range dependencies and leverages Swin Transformer to capture fine-grained local features. In addition, we have developed a self-supervised confidence learning framework that generates pixel-wise reliability maps through photometric consistency constraints, and realizes adaptive fusion of raw depth measurements and network predictions via physics-informed spatial weighting. Meanwhile, we have designed a novel loss function to enhance the accuracy and robustness of depth prediction. Extensive experiments conducted on the TransCG and ClearGrasp datasets validate that CrysFormer++ achieves superior performance compared to existing state-of-the-art approaches, in terms of both visual quality and quantitative metrics. The results validate the effectiveness of CrysFormer++ in handling complex backgrounds, providing a high-precision depth perception solution for robotic grasping of transparent objects.