A Zero-Shot Deep Learning-Supported Sensing System for Crop Seeds and Berries Phenotyping

Hyperspectral imaging, the fusion of vibrational spectral sensing and computer vision, could simultaneously detect the light absorption caused by the molecular vibration transition of chemical bonds as well as the appearance of the target sample. Coupled with artificial intelligence (AI), it is widely used for quality inspection of agricultural products. Sufficient samples with true quality indicators are required for AI model training. An efficient and low-cost approach for single-object extraction in the image is urgently needed. In this study, a zero-shot learning instance segmentation method was proposed for seeds and berries extraction in images of densely arranged samples. The image-annotation pairs for model training were synthesized by combining the objects in a sample pool. Typical segmentation methods including YoloV8-segment, MaskRCNN, and UNet were studied and compared. All these models were trained and optimized using the synthetic datasets and evaluated using real images. Another popular zero-shot learning method segment anything model (SAM) was also studied for comparison. Satisfactory performances were observed on wheat seed extraction in the processing of VIS/NIR hyperspectral images, NIR hyperspectral images, and regular color images. The zero-shot learning-based YoloV8-segment model reached the highest performance on wheat seed segmentation, with mAP50 values from 0.96 to 0.98. Further experiments on image segmentation of lotus seeds, grape berries, and cherry tomatoes also proved its effectiveness. The presented methods could achieve ultrahigh-speed image or spectral database construction of approximate ellipsoid seeds and berries, further promoting the improvement of the efficiency of high-throughput agricultural products phenotyping system.