A lightweight method for apple disease segmentation using multimodal transformer and sensor fusion

To address the challenges of multimodal data fusion, low deployment efficiency, and inadequate recognition robustness in complex environments for fruit tree disease segmentation and severity classification, a multimodal parallel transformer-based framework was proposed for apple disease recognition and grading. This method integrates image data with multi-dimensional environmental sensor information. An image segmentation preprocessing module was incorporated to enhance lesion region representation, while a cross-scale attention mechanism and a frame-wise diffusion module were introduced to improve robustness under challenging backgrounds. Additionally, pruning, quantization, and knowledge distillation techniques were employed to enable lightweight deployment. Experimental results demonstrated that the full model achieved outstanding performance on apple disease recognition tasks, reaching a precision of 0.98, recall of 0.93, F1-score of 0.95, and accuracy of 0.96, surpassing several state-of-the-art methods including Mask R-CNN, SegFormer, and Swin Transformer. After compression, the model size was reduced to 76.4 MB, and computational complexity decreased to 6.1 G, enabling real-time inference speeds of 25.2 FPS and 39.6 FPS on Jetson Xavier and Orin platforms, respectively. Ablation studies confirmed the performance contributions of the segmentation preprocessing, sensor fusion, and diffusion modules, demonstrating the potential of the proposed framework for deployment in resource-constrained agricultural scenarios.