Multimodal weed infestation rate prediction framework for efficient farmland management

Weed, as one of the main hazards of agricultural production, is being widely studied for efficient field management via Multi-spectral sensors. In the field of precision weed control, the weed infestation rate is an important indicator of weed damage, which has been predicted by various methods and attempts to be applied to guide pesticide spraying via UAVs. However, existing prediction methods not only face the problem of scarcity of data types, but most of them also require pixel-level labeling, which makes them difficult to apply practically. It is also challenging to deal with the lack of consistency in multimodal data, which leads to an inability to quantify differences in characteristics between weeds and crops. To address the above problems, we collect a multimodal database (PWMD) of early pepper weeds containing 1495 pairs of visible and infrared images using a UAV and a multispectral camera. Moreover, we further design a multimodal weed infestation rate prediction system (MWPS) to achieve efficient performance in the field. In detail, MWPS implements dual-path generative adversarial learning and a multilevel feature matching module to mitigate modal differences between multimodal images and utilizes a multilayer perceptron model containing dual attention to achieve efficient weed infestation rate prediction. Experimentally validate on our dataset, our proposed framework has a mean square error of 0.12 and a mean absolute error of only 0.09 for the prediction of field weed rates. This study proposes an effective new method for distal field weed management. Code and dataset are available at http://wirps.samlab.cn.