Multimodal Data Fusion by Integrating IoT-Enabled Sensors and Images for Jamun Crop Disease Detection With Machine Learning

Abstract

In agricultural applications, traditional image and sensor-based methods for plant disease prediction face notable limitations. Image-based approaches often struggle with early-stage detection, while sensor-based methods prone to reliability issues due to potential system failures. This study addresses these challenges by integrating complementary data of the Jamun (Syzygium cumini) plant from Internet of Things (IoT)-enabled sensors and mobile-captured images to develop a hybrid machine learning (ML) model for early and accurate plant disease detection. The proposed model combines a multilayer perceptron (MLP) for processing numerical sensor inputs—ambient temperature, soil temperature, relative humidity, soil moisture, and leaf wetness duration—and a convolutional neural network (CNN) for analyzing leaf images labeled as leaf spot, anthracnose, or healthy. Outputs from the MLP and CNN concatenated and processed through an additional MLP to classify plant health effectively. Optimized with hidden layer configurations of 8-16-32-8 for the sensor-data MLP, 16--32-64-128_32-8-4 for the image-data CNN, and 4-3 layers for the final MLP, the model achieves a loss of 1% and an accuracy of 95%, outperforming state-of-the-art methods, such as DenseNet201-support vector machines (SVM) (87.23%) and gray level co-occurrence matrix-SVM (90%). Performance metrics demonstrate high precision (leaf spot: 0.93, anthracnose: 0.93, and healthy: 0.98), recall (leaf spot: 0.92, anthracnose: 0.95, and healthy: 0.96), and F1-scores (leaf spot: 0.92, anthracnose: 0.94, and healthy: 0.97). The model’s deployment on an Amazon Web Services cloud server enables real-time disease detection and classification, making it accessible for practical agricultural use. This sensor and image data integration offers a novel and robust solution to address the limitations of single-modality approaches.