Efficient occlusion avoidance based on active deep sensing for harvesting robots

Abstract

With the increasing shortage of agricultural labor, the development of harvesting robots is becoming more and more urgent. Most of them require vision to locate the target, however, occlusion is common in agricultural environment, which restricts the accuracy of visual target recognition, and even leads to failure in serious cases. The active perception method is an effective means, but how to efficiently find the best observation position remains difficult to avoid the waste of time caused by repeated invalid motion. Targeting these problems, an active deep sensing method is proposed for harvesting clustered and single fruits. First, the region of interest of the target is extracted by a segmentation network, and then the occlusion status of it is obtained by image processing methods. Taking the current observation position as the starting point, the camera is moved within a matrix to form confidence and occlusion rate distribution maps. After establishing a series of occlusion rate and confidence matrix datasets, a designed deep network has been trained, which is used to predict the maximum confidence/minimum occlusion rate position after the current occlusion status is estimated. To verify the reliability of the method, laboratory and field experiments were carried out for apples and clustered tomatoes. After 1000 times of verification, results show that the successful pick/recognition rate is increased by 38.7 %, and the average successful recognition time is 5.2 s, which is 63.1 % and 46.4 % faster than that of a fixed movement method and a simple heuristic method.