Towards transferring skills to flexible surgical robots with programming by demonstration and reinforcement learning

Flexible manipulators such as tendon-driven serpentine manipulators perform better than traditional rigid ones in minimally invasive surgical tasks, including navigation in confined space through key-hole like incisions. However, due to the inherent nonlinearities and model uncertainties, motion control of such manipulators becomes extremely challenging. In this work, a hybrid framework combining Programming by Demonstration (PbD) and reinforcement learning is proposed to solve this problem. Gaussian Mixture Models (GMM), Gaussian Mixture Regression (GMR) and linear regression are used to learn the inverse kinematic model of the manipulator from human demonstrations. The learned model is used as nominal model to calculate the output end-effector trajectories of the manipulator. Two surgical tasks are performed to demonstrate the effectiveness of reinforcement learning: tube insertion and circle following. Gaussian noise is introduced to the standard model and the disturbed models are fed to the manipulator to calculate the actuator input with respect to the task specific end-effector trajectories. An expectation maximization (E-M) based reinforcement learning algorithm is used to update the disturbed model with returns from rollouts. Simulation results have verified that the disturbed model can be converged to the standard one and the tracking accuracy is enhanced.