VTLG: A vision-tactile-language grasp generation method oriented towards task

Abstract

Preceded task-oriented grasp is indispensable for achieving reliable robotic manipulation. Existing task-oriented grasp generation methods typically rely on visual information of the target, deployed on simple parallel-jaw grippers, which often struggle with visual degradation and inadequate grasp reliability and flexibility. In this paper, we propose a task-oriented grasp generation system that integrates external visual and tactile perception on the guidance of textual description. Multimodal encoding consists of two stages: the visual-tactile feature fusion encoding for robot grasp and object spatial perception, and the textual normalization and encoding, followed by spatial perception-semantic feature fusion. We conceive to introduce tactile perception in the pre-contact phase. A visual-tactile fusion method is proposed to combine single-view point cloud with tactile array of pre-contact, partitioning the object surface into multiple contact patches. The vision transformer architecture is employed to gain a spatial representation of the object surface, encoding globally spatial features and implicitly assisting in evaluating the feasibility of grasps across regions by shape learning. To improve the inference effectiveness of the large language model under varying task context, we introduce specifications for textual standardization and migratory comprehension for unknown concepts. The language encoder and principal component analysis are used to encode the given standardized text that follows the above textual generation paradigm. A spatial-semantic feature fusion method is then proposed based on window shift and cross-attention to realize the alignment between task context and object spatial features, ulteriorly preference-based attention on graspable regions. Finally, we present a grasp parameter prediction module based on diffusion model specialized for high-dimensional conditions and generalized hand proprioceptive space, which generates grasp by predicting noise. Experimental results demonstrate that the proposed method outperforms baseline methods requiring the complete object shape, since the mean average precision metrics reaches 72.13%, with an improvement of 1.65%. Each module exhibits performance improvement over conventional methods. Ablation study indicates that the introduction of tactile and text modality improves the metrics by over 3% and 14%. The single-shot success rate for the predicted grasps exceeds 65% on real-world experiments, underscoring the reliability of the proposed system.