A Multimodal Explainable Lightweight Transformer-Based AI Framework for Real-Time Plant Disease Identification using Leaf Images, Environmental Sensors, and Weather Data
P Saranya
Abstract
Purpose This study proposes a multimodal, explainable, and lightweight transformer-based artificial intelligence framework for real-time plant disease identification. The framework is designed to overcome the accuracy-efficiency trade-off of existing convolutional and transformer-based models by jointly leveraging leaf images, environmental sensor readings, and weather data, while remaining interpretable and deployable on resource-constrained edge devices used in the field. Design/Methodology/Approach A multimodal dataset framework is constructed by integrating publicly available leaf image data with synthetically generated environmental sensor observations (soil moisture, humidity, temperature) and weather information (rainfall, wind speed, seasonal indices), used here as a proxy pending access to real paired field data. A lightweight transformer backbone with a cross-modal attention fusion module processes the three modalities jointly, and explainability is incorporated through Grad-CAM and SHAP-based attribution. The proposed model is benchmarked against VGG16, ResNet50, MobileNetV2, EfficientNet-B0, and a standard Vision Transformer using accuracy, precision, recall, F1-score, parameter count, and inference latency. Findings In a preliminary single-run evaluation, the proposed framework achieved 98.5% accuracy with 6.2 million parameters, outperforming image-only baselines (93.4–97.2%). Multimodal fusion improved accuracy by approximately 3.9 percentage points. However, environmental and weather inputs were synthetically generated, and statistical significance was not established. These findings remain preliminary and require validation using real-world multimodal field data. Practical Implications Because of its low parameter count and real-time inference capability, the framework is suitable for deployment on mobile and edge devices used by farmers and agronomists for in-field disease screening. Its explainability outputs support trust and adoption among agricultural extension workers and non-expert end users. Originality/Value This work contributes a novel lightweight cross-modal transformer fusion architecture for plant disease identification that simultaneously optimizes for accuracy, computational efficiency, and interpretability an underexplored combination in the precision agriculture literature, which has largely treated these objectives separately.