Adding Object Manipulation Capabilities to Social Robots by using 3D and RGB Cameras Data

This paper outlines the design and implementation of novel object manipulation for a social robot, here Pepper by SoftBank Robotics. It is primarily designed for verbal interaction and has therefore not been equipped with object manipulation capabilities. The proposed routine exploits the built-in RGB and 3D cameras. First, semantic segmentation based on the Mini-YOLOv3 neural network is run on the RGB image. Next, 3D sensor data are used to position the hand over the object, implementing a novel routine to grab the object and to scan it for recognition purposes. To preserve patient and location sensitive data, the here-proposed architecture operates automatically and offline, running on the robot’s operating system. Experimental results on 370 grabbing processes showed how the manipulation routine achieves a grabbing success rate of up to 96%. They also proved that the success rate remains unaltered if the target object is positioned in a rectangular area of ± 6 cm × ± 3 cm centered in the nominal position provided by an initial positioning grid. The grabbing success rate remains above 80% even if the object to be grabbed is stored with an angle that ranges between 10° and 45° within the above-reported area.