Litchi bunch detection and ripeness assessment using deep learning and clustering with image processing techniques

Abstract

Litchis typically need to be harvested in bunches. The detection of entire litchi bunches and the classification of ripeness are crucial issues for robotic harvesting and are also prerequisites for efficient and non-destructive picking. However, the detection process is complicated by the complex orchard environment and the green colour of litchis, and research on the ripeness grading of litchi bunches is still lacking. To address this issue, this paper proposes a litchi ripeness assessment method combining three components: (1) Litchi-YOLO model, (2) a novel image processing framework, and (3) KGAP-DBSCAN clustering algorithm. The HyCTAS module is embedded into the YOLOv8 model to perform instance segmentation on the litchi fruits in the collected images, obtaining the fruit target points and masks. Then, the KGAP-DBSCAN clustering algorithm automatically clusters the fruit points into litchi bunches by setting the clustering radius ε based on the density of the target points. Ripeness grading of individual fruits is achieved by calculating the proportion of red pericarp and determining the ripeness of litchi bunches based on the agronomic information related to litchi growth. The results show that in terms of detection performance, Litchi-YOLO achieved a precision (P), recall (R), and F₁-score of 95.96 %, 95.69 %, and 95.82 %, respectively, representing improvements of 1.25 %, 6.97 %, and 4.25 % over YOLOv8. In terms of clustering performance, the KGAP-DBSCAN algorithm achieved homogeneity, completeness, and v-measure scores of 0.91, 0.76, and 0.78, respectively, for clustering the fruit coordinate points. The ripeness grading method for individual fruits demonstrated good performance, with a precision of 94.20 % and a recall of 91.91 %. The ripeness of the litchi bunches, calculated from the ripeness parameters of individual fruits and clustering results, meets agronomic requirements. The study assesses the maturity of litchi bunches in a natural environment, assisting the orchard harvesting robot system in determining harvesting decisions.