Bridging Human Intents to Dexterous Grasp: A Language-guided Functional Grasping Approach for Human-centric Smart Manufacturing

Abstract

Enabling collaborative robots to interpret human intent and execute task-specific manipulations constitutes a core requirement for human-centric smart manufacturing, where seamless integration of cognitive flexibility and robotic capability drives production adaptability. Achieving this objective, however, remains constrained when substituting two-fingered grippers with multi-fingered dexterous hands. To overcome these barriers, this paper introduces a novel approach that bridges human intents to dexterous functional grasping. The proposed approach unifies cross-modal 3D affordance segmentation with conditional variational autoencoder-based grasp synthesis, leveraging affordance concepts to align perceptual understanding with task execution while generating diverse, functional grasp poses under multimodal conditioning. To enable real-world deployment with open-vocabulary human instructions, an in-context affordance reasoning module powered by vision-language models parses human instructions and scene context into precise affordance labels. Extensive experiments demonstrate superior performance, achieving 85.5% accuracy in 3D affordance segmentation and low errors in grasp generation. Real-world deployments validate its practicality in collaborative assembly scenarios, such as tool-use and handover, paving the way for more seamless, embodied human-robot collaboration.