Maturity recognition and localisation of broccoli under occlusion based on RGB-D instance segmentation network

Abstract

Selective harvesting robots for broccoli face significant challenges in field operations, where occlusions by leaves and stems, varying maturity stages and lighting interferences greatly affect performance. Addressing the need for a robust network capable of maturity recognition and localisation under various occlusion conditions for spherical crops, OccluInst—a single-stage instance segmentation network based on RGB-D and CNN-Transformer architecture was proposed. The solution is to make full use of visible information and crop characteristics. This model builds a dual-branch cross-modal calibration framework to generate instance-aware kernels and segmentation mask features. The proposed Attention Weight Interactive Fusion Module (AWIF) enhances the fusion efficiency of multi-scale RGB and depth features in complex scenarios, while the designed Adaptive Fusion Ratio Module (AFR) filters out noisy depth data and extracts valuable information to achieve feature alignment. Additionally, the developed Material Awareness Module (MA) highlights critical areas, improving feature extraction for irregular, multi-scale targets. The improved circular boundary anchor box accurately localises broccoli under various levels of occlusion. Ablation studies confirm the effectiveness of each module. OccluInst can swiftly and accurately identify the maturity categories and coordinates of broccoli under different occlusion levels. It achieves a mAP₅₀ of 86.2% and mAR of 83.5%, with an average centre point deviation of 3.68 pixels on images with a resolution of 848 × 480, and a detection speed of 51.4 frames per second, providing a robust visual foundation for selective harvesting robots.