Risk of bias and low reproducibility in meta-analytic evidence from fast-tracked publications during the coronavirus disease 2019 pandemic.

IF 3.8 Q2 MULTIDISCIPLINARY SCIENCES
PNAS nexus Pub Date : 2025-07-29 eCollection Date: 2025-08-01 DOI:10.1093/pnasnexus/pgaf238
Xuerong Liu, Wei Li, Qianyu Zhang, Jingyu Lei, Xiaodi Han, Yaozhi Wang, Chang Shen, Yu Zhan, Yanyan Li, Liping Shi, Jidong Ren, Jingxuan Zhang, Xiaolin Zhang, Yan Wu, Haiping Liao, Lei Xia, Jia Luan, Yue Li, Tatum Madeleine Cummins, Zhengzhi Feng, Chunji Huang, Zhiyi Chen
{"title":"Risk of bias and low reproducibility in meta-analytic evidence from fast-tracked publications during the coronavirus disease 2019 pandemic.","authors":"Xuerong Liu, Wei Li, Qianyu Zhang, Jingyu Lei, Xiaodi Han, Yaozhi Wang, Chang Shen, Yu Zhan, Yanyan Li, Liping Shi, Jidong Ren, Jingxuan Zhang, Xiaolin Zhang, Yan Wu, Haiping Liao, Lei Xia, Jia Luan, Yue Li, Tatum Madeleine Cummins, Zhengzhi Feng, Chunji Huang, Zhiyi Chen","doi":"10.1093/pnasnexus/pgaf238","DOIUrl":null,"url":null,"abstract":"<p><p>The fast-tracked publication of coronavirus disease 2019 (COVID-19)-related meta-analytic evidence has undeniably facilitated rapid public health policymaking; however, concerns are mounting that this publication policy may compromise research quality and scientific integrity. To investigate this, we conducted a meta-research study systematically evaluating risk of bias (ROB), transparency, and reproducibility in pandemic-era meta-analyses synthesizing COVID-19-derived mental health problem epidemics. From 98 identified studies-including data from 18.6 million individuals across 94 countries-we observed significant ROBs in publication, with one new meta-analysis published approximately every 5 days at peak output. Despite apparent sample diversity, nearly half of participants were from China, and only 8.9% originated from less economically developed countries. Of these meta-analyses, a substantial proportion (70.6%) showed discrepancies between Preferred Reporting Items for Systematic Reviews and Meta-Analyses-guided reporting and actual research conducts, while 57.1% exhibited high methodological ROBs due to insufficient data sources and lack of sensitivity analysis. Alarmingly, none achieved full computational reproducibility, and fewer than one-fifth were fully replicable. Furthermore, neither publication in high-impact journals, citation performance, nor fast-track publication mode correlated with lower ROBs that we identified above. To address these limitations, we re-estimated global COVID-19-derived mental health epidemics using their individual participant data after minimizing identified ROBs. Our recalibrated meta-analytic findings provide more reliable benchmarks for understanding the pandemic's mental health impact. This study demonstrated that rigorous methodology and scientific integrity must remain central priorities-even under urgent, crisis-driven conditions-establishing a foundation for transparent, reproducible, and unbiased global mental health surveillance during public health emergencies.</p>","PeriodicalId":74468,"journal":{"name":"PNAS nexus","volume":"4 8","pages":"pgaf238"},"PeriodicalIF":3.8000,"publicationDate":"2025-07-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12342171/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"PNAS nexus","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1093/pnasnexus/pgaf238","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/8/1 0:00:00","PubModel":"eCollection","JCR":"Q2","JCRName":"MULTIDISCIPLINARY SCIENCES","Score":null,"Total":0}
引用次数: 0

Abstract

The fast-tracked publication of coronavirus disease 2019 (COVID-19)-related meta-analytic evidence has undeniably facilitated rapid public health policymaking; however, concerns are mounting that this publication policy may compromise research quality and scientific integrity. To investigate this, we conducted a meta-research study systematically evaluating risk of bias (ROB), transparency, and reproducibility in pandemic-era meta-analyses synthesizing COVID-19-derived mental health problem epidemics. From 98 identified studies-including data from 18.6 million individuals across 94 countries-we observed significant ROBs in publication, with one new meta-analysis published approximately every 5 days at peak output. Despite apparent sample diversity, nearly half of participants were from China, and only 8.9% originated from less economically developed countries. Of these meta-analyses, a substantial proportion (70.6%) showed discrepancies between Preferred Reporting Items for Systematic Reviews and Meta-Analyses-guided reporting and actual research conducts, while 57.1% exhibited high methodological ROBs due to insufficient data sources and lack of sensitivity analysis. Alarmingly, none achieved full computational reproducibility, and fewer than one-fifth were fully replicable. Furthermore, neither publication in high-impact journals, citation performance, nor fast-track publication mode correlated with lower ROBs that we identified above. To address these limitations, we re-estimated global COVID-19-derived mental health epidemics using their individual participant data after minimizing identified ROBs. Our recalibrated meta-analytic findings provide more reliable benchmarks for understanding the pandemic's mental health impact. This study demonstrated that rigorous methodology and scientific integrity must remain central priorities-even under urgent, crisis-driven conditions-establishing a foundation for transparent, reproducible, and unbiased global mental health surveillance during public health emergencies.

2019冠状病毒病大流行期间快速追踪出版物的荟萃分析证据存在偏倚风险和低可重复性。
不可否认,与2019冠状病毒病(COVID-19)相关的元分析证据的快速发表促进了快速的公共卫生政策制定;然而,越来越多的人担心,这种出版政策可能会损害研究质量和科学诚信。为了调查这一点,我们进行了一项荟萃研究,系统地评估了大流行时期综合covid -19衍生精神健康问题流行病的荟萃分析的偏倚风险(ROB)、透明度和可重复性。从98项已确定的研究(包括来自94个国家的1860万人的数据)中,我们观察到出版物中存在显著的罗伯现象,在高峰产出时大约每5天发表一项新的荟萃分析。尽管样本多样性明显,但近一半的参与者来自中国,只有8.9%的参与者来自经济欠发达国家。在这些荟萃分析中,很大一部分(70.6%)显示系统评价和荟萃分析指导报告的首选报告项目与实际研究行为之间存在差异,而57.1%由于数据源不足和缺乏敏感性分析而表现出较高的方法学罗伯。令人担忧的是,没有一个达到完全的计算可重复性,只有不到五分之一的研究是完全可复制的。此外,在高影响力期刊上发表论文、引用绩效和快速出版模式都与我们上面提到的较低的罗伯无关。为了解决这些局限性,我们在最大限度地减少已确定的罗伯后,使用个体参与者数据重新估计了全球covid -19衍生的心理健康流行病。我们重新校准的荟萃分析结果为了解大流行对心理健康的影响提供了更可靠的基准。这项研究表明,严格的方法和科学的完整性必须仍然是核心优先事项——即使在紧急的、危机驱动的条件下——为公共卫生紧急情况下透明、可重复和公正的全球精神卫生监测奠定基础。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
CiteScore
1.80
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信