{"title":"Risk of bias and low reproducibility in meta-analytic evidence from fast-tracked publications during the coronavirus disease 2019 pandemic.","authors":"Xuerong Liu, Wei Li, Qianyu Zhang, Jingyu Lei, Xiaodi Han, Yaozhi Wang, Chang Shen, Yu Zhan, Yanyan Li, Liping Shi, Jidong Ren, Jingxuan Zhang, Xiaolin Zhang, Yan Wu, Haiping Liao, Lei Xia, Jia Luan, Yue Li, Tatum Madeleine Cummins, Zhengzhi Feng, Chunji Huang, Zhiyi Chen","doi":"10.1093/pnasnexus/pgaf238","DOIUrl":null,"url":null,"abstract":"<p><p>The fast-tracked publication of coronavirus disease 2019 (COVID-19)-related meta-analytic evidence has undeniably facilitated rapid public health policymaking; however, concerns are mounting that this publication policy may compromise research quality and scientific integrity. To investigate this, we conducted a meta-research study systematically evaluating risk of bias (ROB), transparency, and reproducibility in pandemic-era meta-analyses synthesizing COVID-19-derived mental health problem epidemics. From 98 identified studies-including data from 18.6 million individuals across 94 countries-we observed significant ROBs in publication, with one new meta-analysis published approximately every 5 days at peak output. Despite apparent sample diversity, nearly half of participants were from China, and only 8.9% originated from less economically developed countries. Of these meta-analyses, a substantial proportion (70.6%) showed discrepancies between Preferred Reporting Items for Systematic Reviews and Meta-Analyses-guided reporting and actual research conducts, while 57.1% exhibited high methodological ROBs due to insufficient data sources and lack of sensitivity analysis. Alarmingly, none achieved full computational reproducibility, and fewer than one-fifth were fully replicable. Furthermore, neither publication in high-impact journals, citation performance, nor fast-track publication mode correlated with lower ROBs that we identified above. To address these limitations, we re-estimated global COVID-19-derived mental health epidemics using their individual participant data after minimizing identified ROBs. Our recalibrated meta-analytic findings provide more reliable benchmarks for understanding the pandemic's mental health impact. This study demonstrated that rigorous methodology and scientific integrity must remain central priorities-even under urgent, crisis-driven conditions-establishing a foundation for transparent, reproducible, and unbiased global mental health surveillance during public health emergencies.</p>","PeriodicalId":74468,"journal":{"name":"PNAS nexus","volume":"4 8","pages":"pgaf238"},"PeriodicalIF":3.8000,"publicationDate":"2025-07-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12342171/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"PNAS nexus","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1093/pnasnexus/pgaf238","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/8/1 0:00:00","PubModel":"eCollection","JCR":"Q2","JCRName":"MULTIDISCIPLINARY SCIENCES","Score":null,"Total":0}
引用次数: 0
Abstract
The fast-tracked publication of coronavirus disease 2019 (COVID-19)-related meta-analytic evidence has undeniably facilitated rapid public health policymaking; however, concerns are mounting that this publication policy may compromise research quality and scientific integrity. To investigate this, we conducted a meta-research study systematically evaluating risk of bias (ROB), transparency, and reproducibility in pandemic-era meta-analyses synthesizing COVID-19-derived mental health problem epidemics. From 98 identified studies-including data from 18.6 million individuals across 94 countries-we observed significant ROBs in publication, with one new meta-analysis published approximately every 5 days at peak output. Despite apparent sample diversity, nearly half of participants were from China, and only 8.9% originated from less economically developed countries. Of these meta-analyses, a substantial proportion (70.6%) showed discrepancies between Preferred Reporting Items for Systematic Reviews and Meta-Analyses-guided reporting and actual research conducts, while 57.1% exhibited high methodological ROBs due to insufficient data sources and lack of sensitivity analysis. Alarmingly, none achieved full computational reproducibility, and fewer than one-fifth were fully replicable. Furthermore, neither publication in high-impact journals, citation performance, nor fast-track publication mode correlated with lower ROBs that we identified above. To address these limitations, we re-estimated global COVID-19-derived mental health epidemics using their individual participant data after minimizing identified ROBs. Our recalibrated meta-analytic findings provide more reliable benchmarks for understanding the pandemic's mental health impact. This study demonstrated that rigorous methodology and scientific integrity must remain central priorities-even under urgent, crisis-driven conditions-establishing a foundation for transparent, reproducible, and unbiased global mental health surveillance during public health emergencies.