End-to-end stereo matching network with two-stage partition filtering for full-resolution depth estimation and precise localization of kiwifruit for robotic harvesting

Abstract

Full-resolution depth estimation within operational space of robotic arms and accurate localization of kiwifruits is very important for automated harvesting. Depth estimation is expected to be accurate and full-resolution while current depth estimation methods are susceptible to depth missing due to occlusion and uneven illumination. And depth estimation mostly focuses on fruit localization, while obstacles such as branches and wires, which can affect harvesting strategy, have not been considered. This paper localized kiwifruits based on bounding boxes output by YOLOv8m and full-resolution depth from an end-to-end stereo matching network, i.e., LaC-Gwc Net, which was trained after generating a stereo matching dataset by proposing a two-stage partition filtering algorithm. Results showed that LaC-Gwc Net achieved an end-point error (EPE) of 3.8 pixels, which means that accurate depth estimation can also be achieved for thin obstacles such as the branches and the wires. Additionally, YOLOv8m obtained acceptable results in detecting kiwifruits and their calyxes, reaching mean average precision (mAP) of 93.1% and detection speed of 7.0 ms. The methodology obtained only kiwifruit localization error of 4.0 mm on the Z-axis, which meets requirements of robotic harvesting. Furthermore, this study considered the localization of obstacles in kiwifruit orchards, providing high-precision full-resolution depth estimation for agricultural harvesting robots.