FruitQuery: A lightweight query-based instance segmentation model for in-field fruit ripeness determination

Accurate fruit instance segmentation at different ripeness stages is critical for developing autonomous harvesting robots, particularly given the unstructured in-field conditions. In this paper, we combine two in-field fruit datasets of peaches and strawberries for multiple ripeness stages determination, and propose a lightweight query-based instance segmentation model named FruitQuery.

The combined dataset contains 3 peach ripeness stages and 4 strawberry ripeness stages, covering various unstructured conditions of two popular fruits. The model FruitQuery consists of three parts: a backbone, a pixel decoder and Transformer decoders. Efficient multi-head self-attention modules are introduced to the backbone to reduce computational overhead, and a pyramid pooling module is added to the pixel decoder to enhance multi-scale feature fusion. Transformer decoders are then applied to learn a fixed number of queries from features and generate instance masks, avoiding postprocessing like non-maximum suppression. FruitQuery runs in an end-to-end way and incorporates the convolution and Transformer to capture fine-grained features related to different fruits at different ripeness stages.

Extensive experiments on the combined fruit dataset demonstrate that our FruitQuery achieves the highest average precision of 67.02 with only 14.08M parameters, outperforming 13 state-of-the-art models with 33 variants. It is noted that FruitQuery surpasses three series of YOLO (v8, v9 and v10) by a large margin. Ablation studies and visualizations also show its robust feature extraction with fewer parameter usage, indicating that the query-based design is effective in localizing fruit. These results highlight FruitQuery's compelling balance between segmentation performance and model size, offering the potential for in-field application.