Development of Autonomous Robotic Bin Picking System Using Convolutional Neural Network (CNN) Initially Trained by Human Skills

Smart Manufacturing (SM) emphasizes autonomous self-adoption and decision making, which is possible by the aid of information technology such as big data, sensors, and machine learning techniques. Picking objects autonomously by industrial robots from cluttered bins (Bin picking) is one of topics that the technologies could be applied to manufacturing processes, especially in flexible input and output logistics. One of the methods is to analyze 3D point clouds from depth sensors, and are matched to the geometry model to calculate possible robot posture, which required heavy calculation and complex algorithm to handle the point clouds. Another method is to train neural networks from reinforced learning, however it requires huge amount of trials and trainings to establish the model, starting with failures. In this paper, a convolutional neural network (CNN) model was initially trained from human skills, and it was trained by itself to improve the job accuracy. In the initial stage, an operator selected a block with a depth image from a Lidar sensor by their intuition that a block can be picked up by a robot. The robot tried to pick up the block, and the image of block with the result of the trial by the robot was recorded. CNN was trained after collecting 500 datasets by the operator. Next, in the self-learning stage, the system automatically tried to pick up candidate blocks from the CNN’s prediction. Collected data during the trial was utilized to gradually train the CNN model. The result shows that the job accuracy was 39% with initial CNN, and improved by 71% after 2,000 trials by self-learning step. The collaboration between human and autonomy would enable to apply the system in shop floors by reduced time, simple development, and improved pick-up accuracy.