STANet-TLA: leveraging deep learning and prior knowledge for large-scale soybean breeding plot segmentation and high-yielding variety screening from UAV time-series data

Abstract

High-yielding varieties screening is essential for food security, which requires the monitoring of canopy growth, the extraction of dynamic traits, and the estimation of yield at the variety level. Unmanned Aerial Vehicle (UAV) provides a valuable source of high-resolution spatio-temporal data, which can accelerate plot-level phenotyping and variety screening. However, the automatic extraction of breeding plot boundaries from UAV images is challenging due to complex backgrounds, dynamic canopies, and varying row and plot intervals. In this study, we introduce a novel method called STANet-TLA for breeding plot extraction to screen high-yielding varieties. STANet-TLA leverages a self-designed spatio-temporal feature alignment network (STANet) for semantic segmentation and a prior knowledge-constrained traction line aggregation method (TLA) for instance segmentation. To evaluate our model, we constructed a comprehensive dataset named SoyUAV, which includes 21,319 images of more than 977 genotypes at almost all growth stages. The results demonstrated that: (1) STANet achieved an intersection over union (IoU) of 85.43 % and an F1-score (F1) of 91.89% for canopy semantic segmentation, outperforming eight state-of-the-art deep learning networks with average improvements of 5.80 % in IoU and 4.65 % in F1. Based on the semantic segmentation results, TLA achieved an IoU of 93.31 % and an F1 of 95.13 % for plot instance segmentation; (2) STANet demonstrated effective transferability across different years, locations, and data types, achieving high IoU scores of 88.22%, 89.53%, and 79.16%, respectively. STANet-TLA was suitable for plot instance segmentation with different planting designs; (3) The accuracy of high-yielding varieties screening was 60 % using Random Forest with static phenotypes in the plots obtained by STANet-TLA segmentation. This accuracy was improved to 70.59 % and 75 % when incorporating time-series and dynamic-fitting phenotypes, respectively. Our datasets and models are publicly available, which we believe will significantly facilitate advanced UAV-based plant phenotyping and widespread large-scale breeding applications.