Multimodal-information-based optimized agricultural prescription recommendation system of crop electronic medical records

Multimodal Crop Electronic Medical Records (CEMRs) contain complex information, including disease symptoms, crop conditions, environmental factors, and diagnostic prescriptions, making them crucial for intelligent prescription recommendations. However, effectively integrating complementary features from different CEMRs modalities has remained a key challenge. Current CEMRs research primarily focuses on unimodal data, and simplistic approaches like feature concatenation struggle to achieve in-depth cross-modal interactions. This study introduces a novel agricultural prescription recommendation model (named AgriPR) based on cross-modal multi-layer feature fusion. The model initially employs task-adaptive pre-trained BERT (TA-BERT) and ConvNeXt to encode text and image unimodal features respectively. Subsequently, it utilizes Bilinear Attention Networks (BAN) to bilinear features and combines them with bimodal encoding features for a multilayer fusion representation. Finally, a dual-layer Transformer performs re-interaction to emphasize key fused features, resulting in precise prescription recommendations. To evaluate AgriPR, we constructed a real CEMRs dataset containing 13 prescription categories from Beijing Plant Clinic. Experimental results demonstrate that AgriPR achieves outstanding performance, with a classification accuracy of 98.88 %, surpassing state-of-the-art models. Furthermore, the study compares and analyzes 8 encoder combinations, 6 feature fusion strategies, and 6 network layer configurations, highlighting the model's design advantages. Lastly, the model's adaptability was also tested with incomplete modality inputs (text-only or image-only) and missing information inputs (e.g., crop, environment, symptoms). The findings confirm AgriPR's practical applicability, providing a high-performance solution for agricultural management systems.