Towards instructional collaborative robots: From video-based learning to feedback-adapted instruction

Abstract

Collaborative robots (cobots) enhanced by artificial intelligence (AI) are enabling intelligent, human-centric manufacturing environments. These dynamic settings require cobots with cognitive intelligence, i.e., capabilities covering perception, learning, decision-making, and adaptation. Such intelligence enables proactive collaboration that integrates bidirectional instructional and cooperative competencies. However, while extensive research has focused on improving the performance of robot collaborative skills, systematic investigations into the instructional capabilities of cobots remain notably limited. To lay the technological foundation for addressing this gap, this survey adopts a multimodal perspective to review three essential aspects of this field: (1) robot learning from video (LfV) for instructional capabilities acquisition, (2) robot-guided instruction methodologies, and (3) feedback-driven adaptation. We present a systematic review of technologies for representing human actions, object states, and human-object interactions (HOI), with a particular focus on multimodal data sources from video. Furthermore, we analyze diverse instructional strategies, including visual guidance, auditory directives, robot-performed actions, emphasizing their effectiveness in robot-guided instruction. A significant focus is placed on feedback-driven adaptation mechanisms, which enable cobots to dynamically refine their instructional capabilities based on user feedback. We identify key challenges such as environmental complexity, user variability, real-time processing constraints, and trust-building requirements, while also highlighting emerging opportunities in multimodal integration, AI-powered robots, and collaborative learning systems. Finally, we underscore the transformative potential of instructional cobots in smart manufacturing and emphasize the necessity for further research.