OVGrasp: Target-oriented open-vocabulary robotic grasping in clutter

Abstract

Robotic recognizing and grasping of novel-category objects in cluttered environments presents a significant challenge due to occlusions and complex object arrangements. In addition, the ability to integrate language instructions is crucial for obtaining target object. In this work, we propose OVGrasp, an open-vocabulary grasping framework that seamlessly integrates vision and language to enhance robotic manipulation capabilities. Our approach leverages a unified integration of pretrained vision-language and grasping models, incorporates cross-modality alignment modules to enhance visual-linguistic perception, and uses a multi-scale voxel based point cloud representation for precise grasp-pose estimation in cluttered environments. By jointly modeling vision, language, and action, OVGrasp eliminates the reliance on predefined object labels and handcrafted rules, enabling more adaptable and efficient grasping. Extensive experiments in both simulation and real-world settings demonstrate that our method can achieve better task success rate by less times of motion, outperforming state-of-the-art methods under open-vocabulary language instructions in cluttered scenarios.