3D positioning of Camellia oleifera fruit-grabbing points for robotic harvesting

Abstract

Camellia oleifera is an oilseed crop with high economic value. The short optimum harvest period and high labour costs of C. oleifera harvesting have prompted research on intelligent robotic harvesting. This study focused on the determination of grabbing points for the robotic harvesting of C. oleifera fruits, providing a basis for the decision making of the fruit-picking robot. A relatively simple 2D convolutional neural network (CNN) and stereoscopic vision replaced the complex 3D CNN to realise the 3D positioning of the fruit. Apple datasets were used for the pretraining of the model and knowledge transfer, which shared a certain degree of similarity to C. oleifera fruit. In addition, a fully automatic coordinate conversion method has been proposed to transform the fruit position information in the image into its 3D position in the robot coordinate system. Results showed that the You Only Look Once (YOLO)v8x model trained using 1012 annotated samples achieved the highest performance for fruit detection, with mAP₅₀ of 0.96 on the testing dataset. With knowledge transfer based on the apple datasets, YOLOv8x using few-shot learning realised a testing mAP₅₀ of 0.95, reducing manual annotation. Moreover, the error in the 3D coordinate calculation was lower than 2.1 cm on the three axes. The proposed method provides the 3D coordinates of the grabbing point for the target fruit in the robot coordinate system, which can be transferred directly to the robot control system to execute fruit-picking actions. This dataset was published online to reproduce the results of this study.