Positioning of mango picking point using an improved YOLOv8 architecture with object detection and instance segmentation

Positioning of mango picking points is a crucial technology for the realisation of automated robotic mango harvesting. Herein, this study reported a visualised end-to-end system for mango picking point positioning using improved YOLOv8 architecture with object detection and instance segmentation, as well as an algorithm of picking point positioning. At first, the improved YOLOv8n model, incorporating the BiFPN structure and the SPD-Conv module, was utilised to enhance the detection performance of mango fruits and stems. This model achieved a detection precision of 98.9% in fruits and 97.1% in stems, with recall of 99.5% and 94.6% respectively. Then, the YOLOv8n-seg model was used for segment the stem ROI (Region of interest), leading to 81.85% in MIoU and 88.69% in mPA. Finally, a skeleton line of the stem region was obtained on the basis of the segmentation image, and a picking point positioning algorithm was developed to determine the coordinates of the optimal picking point. Subsequently, the positioning success rate of coordinates, absolute errors, and relative errors were calculated by comparing the automatic positioned coordinates with the manually positioned stem region. Experimental results indicated that this study achieved an average positioning success rate of 92.01%, with an average absolute error of 4.93 pixels and an average relative error of 13.11%. Additionally, the average processing time for processing 640 images using the picking point positioning system is 72.75 ms. This study demonstrates the reliability and effectiveness of positioning mango picking points, laying the technological basis for the automated harvesting of mango fruits.