PheMuT: A phenology-informed, multi-modal time-series model for strawberry yield forecasting

Abstract

Accurate yield forecasting is crucial in optimizing resource management and decision-making processes in agriculture, particularly in crops such as strawberries, which require precise predictions due to their rapid and continuous ripening cycles. This study introduces PheMuT, a novel phenology-informed, multi-modal time-series model that integrates visual and meteorological data streams to enhance strawberry yield forecasting. The proposed method employs advanced computer vision techniques, including two YOLOv11 detectors, an optimized ByteTrack tracker, Segment Anything (SAM), and Depth Anything v2 (DAv2), for precise fruit detection, canopy, and volume estimation. Concurrently, high-frequency weather data are processed using a self-supervised autoregressive Temporal Convolutional Network (TCN), resulting in concise and informative weather embeddings. These visual and weather features are fused within an LSTM-based model to produce weekly yield forecasts. PheMuT was validated using two strawberry cultivars at a Florida research facility over two consecutive seasons. Results indicated that PheMuT improved forecasting accuracy, reducing mean absolute error (MAE) by 10.7%, root mean squared error (RMSE) by 12.5%, and mean absolute percentage error (MAPE) by 18.6% compared to baseline manual methods. Additionally, the model exhibited a notable improvement of 17.2% in the coefficient of determination (R²). PheMuT offers an efficient, automated framework for yield forecasting. Code and data are available at https://github.com/Sycamorers/PheMuT. The full datasets used in this study are available from the authors upon request.