Visual Question Answer Model Based on Crop Diseases External Knowledge for Smart Agriculture

Abstract

Current crop disease VQA models primarily focus on object counting and detection. However, accurately identifying various disease stages and determining control measures require additional knowledge beyond images, including information about control methods and pathogen details. To address this, the VQA dataset relies on the images and questions to retrieve relevant external knowledge. To realize the VQA task of crop diseases external knowledge, we construct the Visual Question Answer Model Based on Crop Diseases External Knowledge for Smart Agriculture (CDEK). CDEK integrates two categories of external knowledge on 66 common dicotyledonous crop diseases by utilizing large language models and agricultural knowledge repositories to enhance knowledge retrieval. This integration enhances the richness of external knowledge repositories. Enhancing fine-grained image understanding in CDEK through the utilization of Stack Self-Attention (SSA), utilising Cross-Attention and contrastive learning of two external knowledge, with a focus on emphasizing image-related semantic information during training. Finally, an automatic patrol disease detection robot is constructed based on Tensor Processing Unit (TPU) devices and the CDEK model. CDEK achieves an accuracy of 61.7% on the publicly available dataset OK-VQA, surpassing the previous state-of-the-art by 5.1%. Furthermore, we construct the OKiCD-VQA dataset for crop diseases external knowledge and achieve an accuracy of 89.36% using CDEK. A series of ablation experiments are conducted on various modules, the effectiveness of CDEK is demonstrated through extensive experimentation. Contributing solutions to the sustainable development of smart agriculture.