Xing Xing, Jianan Zhu, Linyu Shi, Chang Xu, Lifeng Lin
{"title":"Assessment of inverse publication bias in safety outcomes: an empirical analysis.","authors":"Xing Xing, Jianan Zhu, Linyu Shi, Chang Xu, Lifeng Lin","doi":"10.1186/s12916-024-03707-2","DOIUrl":null,"url":null,"abstract":"<p><strong>Background: </strong>The aims of this study were to assess the presence of inverse publication bias (IPB) in adverse events, evaluate the performance of visual examination, and explore the impact of considering effect direction in statistical tests for such assessments.</p><p><strong>Methods: </strong>We conducted a cross-sectional study using the SMART Safety, the largest dataset for evidence synthesis of adverse events. The visual assessment was performed using contour-enhanced funnel plots, trim-and-fill funnel plots, and sample-size-based funnel plots. Two authors conducted visual assessments of these plots independently, and their agreements were quantified by the kappa statistics. Additionally, IPB was quantitatively assessed using both the one- and two-sided Egger's and Peters' tests.</p><p><strong>Results: </strong>In the SMART Safety dataset, we identified 277 main meta-analyses of safety outcomes with at least 10 individual estimates after dropping missing data. We found that about 13.7-16.2% of meta-analyses exhibited IPB according to the one-sided test results. The kappa statistics for the visual assessments roughly ranged from 0.3 to 0.5, indicating fair to moderate agreement. Using the one-sided Egger's test, 57 out of 72 (79.2%) meta-analyses that initially showed significant IPB in the two-sided test changed to non-significant, while the remaining 15 (20.8%) meta-analyses changed from non-significant to significant.</p><p><strong>Conclusions: </strong>Our findings provide supporting evidence of IPB in the SMART Safety dataset of adverse events. They also suggest the importance of researchers carefully accounting for the direction of statistical tests for IPB, as well as the challenges of assessing IPB using statistical methods, especially considering that the number of studies is typically small. Qualitative assessments may be a necessary supplement to gain a more comprehensive understanding of IPB.</p>","PeriodicalId":9188,"journal":{"name":"BMC Medicine","volume":null,"pages":null},"PeriodicalIF":7.0000,"publicationDate":"2024-10-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11515227/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"BMC Medicine","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1186/s12916-024-03707-2","RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"MEDICINE, GENERAL & INTERNAL","Score":null,"Total":0}
引用次数: 0
Abstract
Background: The aims of this study were to assess the presence of inverse publication bias (IPB) in adverse events, evaluate the performance of visual examination, and explore the impact of considering effect direction in statistical tests for such assessments.
Methods: We conducted a cross-sectional study using the SMART Safety, the largest dataset for evidence synthesis of adverse events. The visual assessment was performed using contour-enhanced funnel plots, trim-and-fill funnel plots, and sample-size-based funnel plots. Two authors conducted visual assessments of these plots independently, and their agreements were quantified by the kappa statistics. Additionally, IPB was quantitatively assessed using both the one- and two-sided Egger's and Peters' tests.
Results: In the SMART Safety dataset, we identified 277 main meta-analyses of safety outcomes with at least 10 individual estimates after dropping missing data. We found that about 13.7-16.2% of meta-analyses exhibited IPB according to the one-sided test results. The kappa statistics for the visual assessments roughly ranged from 0.3 to 0.5, indicating fair to moderate agreement. Using the one-sided Egger's test, 57 out of 72 (79.2%) meta-analyses that initially showed significant IPB in the two-sided test changed to non-significant, while the remaining 15 (20.8%) meta-analyses changed from non-significant to significant.
Conclusions: Our findings provide supporting evidence of IPB in the SMART Safety dataset of adverse events. They also suggest the importance of researchers carefully accounting for the direction of statistical tests for IPB, as well as the challenges of assessing IPB using statistical methods, especially considering that the number of studies is typically small. Qualitative assessments may be a necessary supplement to gain a more comprehensive understanding of IPB.
期刊介绍:
BMC Medicine is an open access, transparent peer-reviewed general medical journal. It is the flagship journal of the BMC series and publishes outstanding and influential research in various areas including clinical practice, translational medicine, medical and health advances, public health, global health, policy, and general topics of interest to the biomedical and sociomedical professional communities. In addition to research articles, the journal also publishes stimulating debates, reviews, unique forum articles, and concise tutorials. All articles published in BMC Medicine are included in various databases such as Biological Abstracts, BIOSIS, CAS, Citebase, Current contents, DOAJ, Embase, MEDLINE, PubMed, Science Citation Index Expanded, OAIster, SCImago, Scopus, SOCOLAR, and Zetoc.