Factors affecting deep learning model performance in citizen science–based image data collection for agriculture: A case study on coffee crops

Abstract

Citizen science is an effective approach for collecting extensive data scalable for deep learning, although data quality is debatable. However, few studies have determined the factors associated with data collection that affect model performance and potential sampling bias. This study aims to identify the factors that significantly influence the performance of a deep learning object detection model in agricultural prediction tasks. To do so, we analyzed errors in a You Only Look Once (YOLO v8) model trained for counting the number of coffee cherries in mobile pictures. The model was trained with 436 images taken in Colombia and Peru collected by local farmers as a citizen science approach. We analyzed the prediction errors of the model using 637 additional pictures. We then applied a linear mixed model (LMM) and a decision tree machine learning model to regress the model’s error against predictor variables related to the following categories: photographer influence, geographic location, mobile phone characteristics, picture characteristics, and coffee varieties. Our results show the strong influence of photographer identity and adherence (whether the image collection protocol was followed or not) on model prediction error. Following the protocol can increase model performance from an $R^2$ of 0.48 to 0.73. Additionally, model performance varied significantly depending on photographer identity, with $R^2$ ranging from 0.45 to 0.93. In contrast, factors such as mobile phone characteristics (e.g., frontal camera resolution, flash type, and screen size), using the screen behind the branch to obscure other cherries, coffee varieties, and geographic location did not significantly affect prediction error. These findings demonstrate that data quality in citizen science–based data collection for enhancing model prediction can be achieved through straightforward and comprehensive protocols, customized volunteer training, and regular feedback from experts. Such measures collectively support the robust application of deep learning models in agriculture. Furthermore, this study demonstrated that any mobile device with a camera can contribute to citizen science initiatives, underscoring the potential and scalability of this approach in agricultural research.