Fault tolerant robot programming by demonstration of sorting tasks with industrial objects

The goal of programming by demonstration (PBD) (also known as “learning by demonstration” and “imitation learning”) is for the robot to learn its program from a human demonstrator or teacher, saving time and money compared with traditional robot programming. This paper focuses on programming robots using human pointing gestures to automatically sort objects into bins. The proposed PBD system's software design, algorithms and experimental implementation are presented. Gesturing, speech and graphics facilitate the human-robot interaction. The main novelty of the system is its ability to tolerate human and robot faults. The tolerated human faults include: vague pointing gesture, timeout during pointing, pointing to previously matched class, pointing to previously matched bin, unclassified object found, matching bin not found, and human inside work zone during sorting task. Dropping an object during pick-and-place is the tolerated robot fault. The hardware includes a single color plus depth camera, and a six- axis robotic arm with an electromagnetic gripper. The software runs on a standard PC. The system's ability to deal with human and robot faults was verified using teaching and sorting experiments performed with a set of industrial parts.