Emma G Iorio, Alireza Khanteymoori, Kenneth A Fond, Anastasia V Keller, Lex Maliga Davis, Jan M Schwab, Adam R Ferguson, Abel Torres-Espin, Ralf Watzlawick
{"title":"Effect-Size Discrepancies in Literature Versus Raw Datasets from Experimental Spinal Cord Injury Studies: A CLIMBER Meta-Analysis.","authors":"Emma G Iorio, Alireza Khanteymoori, Kenneth A Fond, Anastasia V Keller, Lex Maliga Davis, Jan M Schwab, Adam R Ferguson, Abel Torres-Espin, Ralf Watzlawick","doi":"10.1089/neur.2024.0038","DOIUrl":null,"url":null,"abstract":"<p><p>Translation of spinal cord injury (SCI) therapeutics from pre-clinical animal studies into human studies is challenged by effect size variability, irreproducibility, and misalignment of evidence used by pre-clinical versus clinical literature. Clinical literature values reproducibility, with the highest grade evidence (class 1) consisting of meta-analysis demonstrating large therapeutic efficacy replicating across multiple studies. Conversely, pre-clinical literature values novelty over replication and lacks rigorous meta-analyses to assess reproducibility of effect sizes across multiple articles. Here, we applied modified clinical meta-analysis methods to pre-clinical studies, comparing effect sizes extracted from published literature to raw data on individual animals from these same studies. Literature-extracted data (LED) from numerical and graphical outcomes reported in publications were compared with individual animal data (IAD) deposited in a federally supported repository of SCI data. The animal groups from the IAD were matched with the same cohorts in the LED for a direct comparison. We applied random-effects meta-analysis to evaluate predictors of neuroconversion in LED versus IAD. We included publications with common injury models (contusive injuries) and standardized end-points (open field assessments). The extraction of data from 25 published articles yielded <i>n</i> = 1841 subjects, whereas IAD from these same articles included <i>n</i> = 2441 subjects. We observed differences in the number of experimental groups and animals per group, insufficient reporting of dropout animals, and missing information on experimental details. Meta-analysis revealed differences in effect sizes across LED versus IAD stratifications, for instance, severe injuries had the largest effect size in LED (standardized mean difference [SMD = 4.92]), but mild injuries had the largest effect size in IAD (SMD = 6.06). Publications with smaller sample sizes yielded larger effect sizes, while studies with larger sample sizes had smaller effects. The results demonstrate the feasibility of combining IAD analysis with traditional LED meta-analysis to assess effect size reproducibility in SCI.</p>","PeriodicalId":74300,"journal":{"name":"Neurotrauma reports","volume":null,"pages":null},"PeriodicalIF":1.8000,"publicationDate":"2024-07-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11271150/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Neurotrauma reports","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1089/neur.2024.0038","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/1/1 0:00:00","PubModel":"eCollection","JCR":"Q3","JCRName":"CLINICAL NEUROLOGY","Score":null,"Total":0}
引用次数: 0
Abstract
Translation of spinal cord injury (SCI) therapeutics from pre-clinical animal studies into human studies is challenged by effect size variability, irreproducibility, and misalignment of evidence used by pre-clinical versus clinical literature. Clinical literature values reproducibility, with the highest grade evidence (class 1) consisting of meta-analysis demonstrating large therapeutic efficacy replicating across multiple studies. Conversely, pre-clinical literature values novelty over replication and lacks rigorous meta-analyses to assess reproducibility of effect sizes across multiple articles. Here, we applied modified clinical meta-analysis methods to pre-clinical studies, comparing effect sizes extracted from published literature to raw data on individual animals from these same studies. Literature-extracted data (LED) from numerical and graphical outcomes reported in publications were compared with individual animal data (IAD) deposited in a federally supported repository of SCI data. The animal groups from the IAD were matched with the same cohorts in the LED for a direct comparison. We applied random-effects meta-analysis to evaluate predictors of neuroconversion in LED versus IAD. We included publications with common injury models (contusive injuries) and standardized end-points (open field assessments). The extraction of data from 25 published articles yielded n = 1841 subjects, whereas IAD from these same articles included n = 2441 subjects. We observed differences in the number of experimental groups and animals per group, insufficient reporting of dropout animals, and missing information on experimental details. Meta-analysis revealed differences in effect sizes across LED versus IAD stratifications, for instance, severe injuries had the largest effect size in LED (standardized mean difference [SMD = 4.92]), but mild injuries had the largest effect size in IAD (SMD = 6.06). Publications with smaller sample sizes yielded larger effect sizes, while studies with larger sample sizes had smaller effects. The results demonstrate the feasibility of combining IAD analysis with traditional LED meta-analysis to assess effect size reproducibility in SCI.
脊髓损伤(SCI)疗法从临床前动物研究转化为人体研究面临着效应大小可变性、不可再现性以及临床前文献与临床文献所用证据不一致的挑战。临床文献重视可重复性,最高级别的证据(1 级)由荟萃分析组成,可在多项研究中重复显示巨大的疗效。相反,临床前文献重视新颖性而非重复性,缺乏严格的荟萃分析来评估多篇文章中效应大小的重复性。在此,我们将修改后的临床荟萃分析方法应用于临床前研究,比较了从已发表文献中提取的效应大小与这些相同研究中的动物个体原始数据。我们将从出版物中报告的数字和图表结果中提取的文献数据(LED)与存放在联邦政府支持的 SCI 数据库中的动物个体数据(IAD)进行了比较。IAD 中的动物组群与 LED 中的相同组群相匹配,以便进行直接比较。我们采用随机效应荟萃分析来评估 LED 与 IAD 中神经转换的预测因素。我们纳入了具有共同损伤模型(挫伤)和标准化终点(开放场地评估)的出版物。从 25 篇已发表的文章中提取的数据得出了 n = 1841 个受试者,而从这些相同的文章中提取的 IAD 数据则得出了 n = 2441 个受试者。我们观察到实验组和每组动物数量的差异、对辍学动物报告的不足以及实验细节信息的缺失。Meta 分析显示,LED 与 IAD 分层的效应大小存在差异,例如,重度损伤在 LED 中的效应大小最大(标准化平均差 [SMD = 4.92]),但轻度损伤在 IAD 中的效应大小最大(SMD = 6.06)。样本量较小的文献产生的效应大小较大,而样本量较大的研究产生的效应较小。结果表明,将IAD分析与传统的LED荟萃分析相结合来评估SCI的效应大小再现性是可行的。