Multimodal information fusion and precision harvesting system for fruit growth driven by flexible optoelectronic sensing and hierarchical attention networks

Abstract

To enhance fruit yield and quality, this study focuses on precise pre-harvest ripeness assessment and early disease detection. Addressing the limitations of conventional methods, we propose a multimodal flexible sensing and deep learning-based evaluation framework. The developed flexible optoelectronic in-situ sensing system integrates spectral (410–940 nm, 18 channels) and impedance (100  Hz–10  kHz) detection, allowing conformal attachment to mango surfaces for nondestructive monitoring throughout the growth cycle while collecting spectral, impedance, and physicochemical data. The proposed 1DCNN-ATT-BiLSTM-ATT network employs independent branches to extract local features from each modality, followed by attention mechanisms and temporal modelling for comprehensive feature fusion, achieving 97.5 % accuracy on test sets. Field experiments reveal systematic variations in soluble solid content (SSC), moisture content (MC), and optoelectronic signals during ripening. Correlation and Granger causality analyses underscore the necessity of multimodal fusion. This system supports intelligent harvesting and precision monitoring, advancing agricultural practices toward greater efficiency and sustainability while establishing a technical paradigm for precision agriculture. Future work will focus on improving environmental robustness and cross-cultivar applicability.