Visual control of a cable-driven flexible robotic arm with a spinal structure based on video understanding

Flexible robotic arms have higher degrees of freedom, making them increasingly popular in tasks such as grasping in narrow environments. However, their high-compliance characteristics and occlusion in the environment pose considerable challenges in the precise control consistent with human requirements. This study combines vision-based knowledge representation with a large language model to help a cable-driven flexible robotic arm with a spinal structure better understand human intentions and mimic human actions. In particular, a visual servo system closely coupled with the flexible robotic arm is designed, which can effectively reduce the impact of occlusion on visual positioning. In a narrow experimental environment, the recognition accuracy of the coupled visual dynamic adjustment system improved by 34.8% compared with relying solely on visual recognition from the end of the arm, and by 28.7% compared with using the external camera visual recognition alone. Subsequently, aimed to perform fine manipulations of the flexible robotic arm, a data-driven nonlinear modeling method is proposed and a coarse-to-fine visual grasping control system is designed. Experiments across eight task scenarios validate the precise control and interactivity of the system in narrow environments using a flexible robotic arm with a spinal structure.