Foundation model-based apple ripeness and size estimation for selective harvesting

Harvesting is a critical task in the tree fruit industry, demanding extensive manual labor and substantial costs, and exposing workers to potential hazards. Recent advances in automated harvesting offer a promising solution by enabling efficient, cost-effective, and ergonomic fruit picking within tight harvesting windows. However, existing harvesting technologies often indiscriminately harvest all visible and accessible fruits, including those that are unripe or undersized. This study introduces a novel foundation-model-based framework for efficient apple ripeness and size estimation. Specifically, we curated two public RGBD-based Fuji apple image datasets, integrating expanded annotations for ripeness (“Ripe” vs. “Unripe”) based on fruit color and image capture dates. The resulting comprehensive dataset, Fuji-Ripeness-Size Dataset, includes 4,027 images and 16,257 annotated apples with ripeness and size labels. To the best of our knowledge, this is the first published dataset on apples with ripeness and size annotations. Leveraging Grounding-DINO, a foundation-model-based object detector, we achieved robust apple detection and ripeness estimation, with mean Average Precision being 72.8, outperforming other state-of-the-art models in the evaluation on our dataset. Additionally, we developed six size estimation algorithms, made a comprehensive comparison using box-plots, and identified the best algorithm with lowest error and variation. The Fuji-Ripeness-Size Dataset and the apple detection and size estimation algorithms are made publicly available¹, which provides valuable benchmarks for future studies in automated and selective harvesting.