TomPhenoNet: A multi-modal fusion and multi-task learning network model for monitoring growth parameters of dwarf tomatoes

Abstract

Dwarf tomatoes, with high edible and ornamental value, require monitoring multiple growth parameters to balance yield and aesthetics. While deep learning has been widely applied in phenotype monitoring, most studies focus on individual growth parameters, overlooking intrinsic relationships. To simultaneously monitor multiple growth parameters across the entire growth stage and different cultivars, this study develops a multi-modal multi-task phenotype monitoring network for dwarf tomatoes (TomPhenoNet). The network model utilizes top-view RGB-D images to evaluate four key growth parameters: height, leaf area, fresh weight, and the number of red fruits. TomPhenoNet generates mask images, fruit detection features, and the number of detected fruits based on RGB images. By fusing RGB-D images, mask images, and fruit detection features, and introducing the cross-stitch network, the network predicts plant height, leaf area, and fresh weight. The predicted values are further used to generate the dynamic occlusion coefficient, adjusting the number of detected fruits to accurately predict the number of red fruits. Results reveal that TomPhenoNet achieves high prediction performances, with R² values of 0.828, 0.930, 0.945, and 0.881 for plant height, leaf area, fresh weight, and the number of red fruits, respectively. Ablation experiments show that the cross-stitch network and fruit detection features improve the prediction performances of growth parameters, with TomPhenoNet combining both modules performing best. Feature importance analysis indicates the network model captures plant growth characteristics and corrects the impact of leaf occlusion from the top view. This study promotes accurate tomato monitoring and provides data support for optimizing cultivation strategies.