Quality assessment of meta-analyses on soil organic carbon

Abstract. Soil organic carbon (SOC) plays a vital role in the global carbon cycle and is a potential sink for carbon dioxide. Agricultural management practices can support carbon sequestration and, therefore, offer potential removal strategies whilst also improving overall soil quality. Meta-analysis allows one to summarize results from primary articles by calculating an overall effect size and to reveal the source of variation across studies. The number of meta-analyses published in the field of agriculture is continuously rising. At the same time, more and more articles refer to their synthesis work as a meta-analysis, despite applying less than rigorous methodologies. As a result, poor-quality meta-analyses are published and may lead to questionable conclusions and recommendations to scientists, policymakers, and farmers. This study aims at quantitatively analyzing 31 meta-analyses, published between the years of 2005 and 2020, studying the effects of different management practices on SOC. We compiled a set of quality criteria suitable for soil and agricultural sciences by adapting existing meta-analytical guidelines from other disciplines. The set is supported by a scoring scheme that allows for a quantitative analysis. The retrieved meta-analyses were structured according to 11 management categories, such as tillage, cover crops, crop residue management, and biochar application, which allowed us to assess the state of knowledge on these categories. Major deficiencies were found in the use of standard metrics for effect size calculation, independence of effect sizes, standard deviation extraction for each study, and study weighting by the inverse of variance. Only 1 out of 31 SOC meta-analyses, which studied the effects of no tillage/reduced tillage compared with conventional tillage, was found to be of high quality. Therefore, improved meta-analyses on the effects of organic agriculture, biochar, fertilization, or crop diversification on SOC are urgently needed. We conclude that, despite efforts over the last 15 years, the quality of meta-analyses on SOC research is still low. Thus, in order for the scientific community to provide high-quality synthesis work and to make advancements in the sustainable management of agricultural soils, we need to adapt rigorous methodologies of meta-analysis as quickly as possible.