Taxonomic hierarchical loss function for enhanced crop and weed phenotyping in multi-task semantic segmentation

Abstract

Herbicide research and development necessitate specific trials to monitor the effects of various herbicide formulations, quantities, and protocols on different plant species and growth stages. These trials are necessary to ensure the safety and efficacy of the developed products. Currently, these tests are conducted manually and assessed visually, making the process time-consuming and labor-intensive. Developing a computer model to characterize species, damage, and growth stages is challenging due to the fine-grained differences between species and damage, significant intra-class variability, and difficulties in manual annotations. Additionally, manually annotated datasets for semantic segmentation are often imperfect. The presence of non-target or unknown species, where only the genus or family is known, complicates the management and scalability of these datasets.

In this work, we propose a new hierarchical loss function, suitable for semantic segmentation tasks, capable to take advantage for the hierarchical taxonomy relationships between species, plant damages and other relationships and thus, reduce the need for annotated data. The proposed loss function support datasets with varying granularity and annotation heterogeneity, including for partial annotations at the pixel level. We validated this loss function using a multi-task semantic segmentation neural network to simultaneously detect plant species and quantify the damage of each species. The proposed hierarchical loss function improves model performance, increasing the F1-Score for species detection from 0.41 to 0.52, for damage detection from 0.23 to 0.28. This enhancement forces the model to learn richer hierarchical representations, enabling the support of heterogeneous and partially annotated scalable datasets, which are common in real-world AI applications.