A multisensor system for observation of user actions in programming by demonstration

Good observation of a manipulation presentation performed by a human teacher is crucial to further processing steps in programming by demonstration in interactive robot programming. This paper outlines a concept of how this can be done using visual and finger measuring sensors. The input sources include: a data glove which classifies several gestures and grasps, an active stereo vision, and a fixed ceiling camera. The hardware used is presented together with the technical concepts of processing and the acquired sensor information fusion, so called elementary cognitive operators. All the sensor sources use time-efficient algorithms, since sensor data must be processed in real-time.