Can Non-Randomised Studies of Interventions Provide Unbiased Effect Estimates? A Systematic Review of Internal Replication Studies.

IF 3 4区 社会学 Q1 SOCIAL SCIENCES, INTERDISCIPLINARY
Hugh Sharma Waddington, Paul Fenton Villar, Jeffrey C Valentine
{"title":"Can Non-Randomised Studies of Interventions Provide Unbiased Effect Estimates? A Systematic Review of Internal Replication Studies.","authors":"Hugh Sharma Waddington,&nbsp;Paul Fenton Villar,&nbsp;Jeffrey C Valentine","doi":"10.1177/0193841X221116721","DOIUrl":null,"url":null,"abstract":"<p><p>Non-randomized studies of intervention effects (NRS), also called quasi-experiments, provide useful decision support about development impacts. However, the assumptions underpinning them are usually untestable, their verification resting on empirical replication. The internal replication study aims to do this by comparing results from a causal benchmark study, usually a randomized controlled trial (RCT), with those from an NRS conducted at the same time in the sampled population. We aimed to determine the credibility and generalizability of findings in internal replication studies in development economics, through a systematic review and meta-analysis. We systematically searched for internal replication studies of RCTs conducted on socioeconomic interventions in low- and middle-income countries. We critically appraised the benchmark randomized studies, using an adapted tool. We extracted and statistically synthesized empirical measures of bias. We included 600 estimates of correspondence between NRS and benchmark RCTs. All internal replication studies were found to have at least \"some concerns\" about bias and some had high risk of bias. We found that study designs with selection on unobservables, in particular regression discontinuity, on average produced absolute standardized bias estimates that were approximately zero, that is, equivalent to the estimates produced by RCTs. But study conduct also mattered. For example, matching using pre-tests and nearest neighbor algorithms corresponded more closely to the benchmarks. The findings from this systematic review confirm that NRS can produce unbiased estimates. Authors of internal replication studies should publish pre-analysis protocols to enhance their credibility.</p>","PeriodicalId":47533,"journal":{"name":"Evaluation Review","volume":"47 3","pages":"563-593"},"PeriodicalIF":3.0000,"publicationDate":"2023-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/e0/ef/10.1177_0193841X221116721.PMC10186563.pdf","citationCount":"4","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Evaluation Review","FirstCategoryId":"90","ListUrlMain":"https://doi.org/10.1177/0193841X221116721","RegionNum":4,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"SOCIAL SCIENCES, INTERDISCIPLINARY","Score":null,"Total":0}
引用次数: 4

Abstract

Non-randomized studies of intervention effects (NRS), also called quasi-experiments, provide useful decision support about development impacts. However, the assumptions underpinning them are usually untestable, their verification resting on empirical replication. The internal replication study aims to do this by comparing results from a causal benchmark study, usually a randomized controlled trial (RCT), with those from an NRS conducted at the same time in the sampled population. We aimed to determine the credibility and generalizability of findings in internal replication studies in development economics, through a systematic review and meta-analysis. We systematically searched for internal replication studies of RCTs conducted on socioeconomic interventions in low- and middle-income countries. We critically appraised the benchmark randomized studies, using an adapted tool. We extracted and statistically synthesized empirical measures of bias. We included 600 estimates of correspondence between NRS and benchmark RCTs. All internal replication studies were found to have at least "some concerns" about bias and some had high risk of bias. We found that study designs with selection on unobservables, in particular regression discontinuity, on average produced absolute standardized bias estimates that were approximately zero, that is, equivalent to the estimates produced by RCTs. But study conduct also mattered. For example, matching using pre-tests and nearest neighbor algorithms corresponded more closely to the benchmarks. The findings from this systematic review confirm that NRS can produce unbiased estimates. Authors of internal replication studies should publish pre-analysis protocols to enhance their credibility.

Abstract Image

Abstract Image

Abstract Image

干预措施的非随机研究能否提供无偏效果估计?内部复制研究的系统回顾。
干预效果的非随机研究(NRS),也称为准实验,为发展影响提供了有用的决策支持。然而,支撑它们的假设通常是不可检验的,它们的验证依赖于经验复制。内部复制研究的目的是通过比较因果基准研究(通常是随机对照试验(RCT))的结果与同时在抽样人群中进行的全国抽样调查(NRS)的结果来做到这一点。我们的目的是通过系统回顾和荟萃分析来确定发展经济学内部重复性研究结果的可信度和普遍性。我们系统地检索了在中低收入国家进行的社会经济干预的随机对照试验的内部复制研究。我们使用一种适应性工具对基准随机研究进行了批判性评价。我们提取并统计合成了偏差的经验度量。我们纳入了600个NRS和基准rct之间的对应估计。所有的内部复制研究都被发现至少对偏倚存在“一些担忧”,其中一些有很高的偏倚风险。我们发现,选择不可观测的研究设计,特别是回归不连续,平均产生的绝对标准化偏差估计近似为零,即等同于随机对照试验产生的估计。但学习行为也很重要。例如,使用预测试和最近邻算法进行匹配更接近基准。该系统综述的结果证实,NRS可以产生无偏估计。内部复制研究的作者应该发表分析前协议,以提高其可信度。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Evaluation Review
Evaluation Review SOCIAL SCIENCES, INTERDISCIPLINARY-
CiteScore
2.90
自引率
11.10%
发文量
80
期刊介绍: Evaluation Review is the forum for researchers, planners, and policy makers engaged in the development, implementation, and utilization of studies aimed at the betterment of the human condition. The Editors invite submission of papers reporting the findings of evaluation studies in such fields as child development, health, education, income security, manpower, mental health, criminal justice, and the physical and social environments. In addition, Evaluation Review will contain articles on methodological developments, discussions of the state of the art, and commentaries on issues related to the application of research results. Special features will include periodic review essays, "research briefs", and "craft reports".
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信