High-performance prediction of soil organic carbon using automatic hyperparameter optimization method in the yellow river delta of China

Using machine learning (ML) and deep learning (DL) models to predict the spatial variability of soil organic carbon (SOC) is crucial for advancing carbon emission reduction strategies. However, inadequate hyperparameter tuning remains a key limitation, reducing the model fitting performance and prediction accuracy. Notably, high-performance models enabled by automatic hyperparameter optimization (AHPO) represent a novel approach to explain the complex relationships between environmental factors and SOC. In this study, we analyzed the prediction performance of ML models, such as gradient boosting decision tree (GBDT) and extreme gradient boosting (XGB), and DL models, including deep forest (DF) and convolutional neural network (CNN). These models were optimized using nature-inspired algorithms (grey wolf optimization (GWO) and hunter-prey optimization (HPO)) and mathematical-approximation algorithms (Bayesian optimization (BO) and tree-structured Parzen estimator (TPE). Furthermore, we derived the linear and nonlinear driving effects of environmental factors (soil, vegetation, texture, climate, and terrain) on SOC. We also identified direct and indirect response pathways using SHapley additive interpretation (SHAP), variogram decomposition (VD), hierarchical partitioning (HP), and structural equation model (SEM). Our results show that prediction models optimized with mathematical approximation algorithms, such as BO-DF (R² = 0.76) and TPE-DF (R² = 0.82), demonstrated the strongest nonlinear fitting ability between environmental factors and SOC. AHPO algorithms significantly improved the prediction performance of DL models, with R² values for the four optimization methods increasing from 0.72 to 0.82. The generalization verification results indicate that the TPE-optimized model demonstrates strong robustness and achieves the highest accuracy (R² > 0.7) for SOC prediction. The AHPO prediction model’s hyperparameter combination achieves a balance between similarity and distinctiveness, where key performance-determining hyperparameters exhibit significant variation (i.e. non-similarity), enabling high-performance SOC predictions. The spatial mapping using the TPE-DF model revealed that areas with high SOC content are primarily concentrated in the southern and northeastern regions of the study area. Moreover, when the model’s prediction accuracy (R²) exceeds 0.75, SHAP analysis identifies SoilAN, SoilAP, SoilAK, TMP, and PRE as the most influential environmental factors driving nonlinear changes in SOC. Similarly, VD and HP analyses highlight a synergistic linear contribution of soil and climate factors, accounting for 99.1 % of the variability in SOC. Interestingly, the path analysis further indicates that regional climate warming leads to surface soil desiccation and salinization, which significantly alters the SOC decomposition environment. High salt stress negatively affects microorganisms and crop root activity, ultimately enhancing SOC accumulation in surface soil. Overall, AHPO-empowered ML and DL methods exhibit strong feasibility for analyzing the response relationship between environmental factors and SOC. Therefore, these methods provide robust support for high-performance and high-precision SOC monitoring across spatial scales.