随机对照试验的不准确预测

IF 3.2 Q1 POLITICAL SCIENCE

Journal of Experimental Political Science Pub Date : 2023-11-22 DOI:10.1017/xps.2023.28

Mats-Philip Ahrenshop, M. Golden, Saad Gulzar, Luke Sonnet

{"title":"随机对照试验的不准确预测","authors":"Mats-Philip Ahrenshop, M. Golden, Saad Gulzar, Luke Sonnet","doi":"10.1017/xps.2023.28","DOIUrl":null,"url":null,"abstract":"We report the results of a forecasting experiment about a randomized controlled trial that was conducted in the field. The experiment asks Ph.D. students, faculty, and policy practitioners to forecast (1) compliance rates for the RCT and (2) treatment effects of the intervention. The forecasting experiment randomizes the order of questions about compliance and treatment effects and the provision of information that a pilot experiment had been conducted which produced null results. Forecasters were excessively optimistic about treatment effects and unresponsive to item order as well as to information about a pilot. Those who declare themselves expert in the area relevant to the intervention are particularly resistant to new information that the treatment is ineffective. We interpret our results as suggesting that we should exercise caution when undertaking expert forecasting, since experts may have unrealistic expectations and may be inflexible in altering these even when provided new information.","PeriodicalId":37558,"journal":{"name":"Journal of Experimental Political Science","volume":null,"pages":null},"PeriodicalIF":3.2000,"publicationDate":"2023-11-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Inaccurate forecasting of a randomized controlled trial\",\"authors\":\"Mats-Philip Ahrenshop, M. Golden, Saad Gulzar, Luke Sonnet\",\"doi\":\"10.1017/xps.2023.28\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"We report the results of a forecasting experiment about a randomized controlled trial that was conducted in the field. The experiment asks Ph.D. students, faculty, and policy practitioners to forecast (1) compliance rates for the RCT and (2) treatment effects of the intervention. The forecasting experiment randomizes the order of questions about compliance and treatment effects and the provision of information that a pilot experiment had been conducted which produced null results. Forecasters were excessively optimistic about treatment effects and unresponsive to item order as well as to information about a pilot. Those who declare themselves expert in the area relevant to the intervention are particularly resistant to new information that the treatment is ineffective. We interpret our results as suggesting that we should exercise caution when undertaking expert forecasting, since experts may have unrealistic expectations and may be inflexible in altering these even when provided new information.\",\"PeriodicalId\":37558,\"journal\":{\"name\":\"Journal of Experimental Political Science\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":3.2000,\"publicationDate\":\"2023-11-22\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of Experimental Political Science\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1017/xps.2023.28\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"POLITICAL SCIENCE\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Experimental Political Science","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1017/xps.2023.28","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"POLITICAL SCIENCE","Score":null,"Total":0}

引用次数: 0

摘要

我们报告了一项在实地进行的随机对照试验预测实验的结果。实验要求博士生、教师和政策实践者预测 (1) 随机对照试验的达标率和 (2) 干预的治疗效果。预测实验随机安排了有关依从性和治疗效果的问题的顺序，并提供了已进行试点实验但结果为空的信息。预测者对治疗效果过于乐观，对项目顺序和试点信息反应迟钝。那些宣称自己是干预措施相关领域专家的人，对治疗无效的新信息尤其抗拒。我们对研究结果的解释是，我们在进行专家预测时应谨慎行事，因为专家们可能会有不切实际的期望，而且即使在获得新信息时，他们也可能不会灵活地改变这些期望。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

查看原文本刊更多论文

Inaccurate forecasting of a randomized controlled trial

We report the results of a forecasting experiment about a randomized controlled trial that was conducted in the field. The experiment asks Ph.D. students, faculty, and policy practitioners to forecast (1) compliance rates for the RCT and (2) treatment effects of the intervention. The forecasting experiment randomizes the order of questions about compliance and treatment effects and the provision of information that a pilot experiment had been conducted which produced null results. Forecasters were excessively optimistic about treatment effects and unresponsive to item order as well as to information about a pilot. Those who declare themselves expert in the area relevant to the intervention are particularly resistant to new information that the treatment is ineffective. We interpret our results as suggesting that we should exercise caution when undertaking expert forecasting, since experts may have unrealistic expectations and may be inflexible in altering these even when provided new information.

求助全文

通过发布文献求助，成功后即可免费获取论文全文。去求助

来源期刊

Journal of Experimental Political Science Social Sciences-Sociology and Political Science

CiteScore

5.10

自引率

8.30%

发文量

期刊介绍： The Journal of Experimental Political Science (JEPS) features cutting-edge research that utilizes experimental methods or experimental reasoning based on naturally occurring data. We define experimental methods broadly: research featuring random (or quasi-random) assignment of subjects to different treatments in an effort to isolate causal relationships in the sphere of politics. JEPS embraces all of the different types of experiments carried out as part of political science research, including survey experiments, laboratory experiments, field experiments, lab experiments in the field, natural and neurological experiments. We invite authors to submit concise articles (around 4000 words or fewer) that immediately address the subject of the research. We do not require lengthy explanations regarding and justifications of the experimental method. Nor do we expect extensive literature reviews of pros and cons of the methodological approaches involved in the experiment unless the goal of the article is to explore these methodological issues. We expect readers to be familiar with experimental methods and therefore to not need pages of literature reviews to be convinced that experimental methods are a legitimate methodological approach. We will consider longer articles in rare, but appropriate cases, as in the following examples: when a new experimental method or approach is being introduced and discussed or when novel theoretical results are being evaluated through experimentation. Finally, we strongly encourage authors to submit manuscripts that showcase informative null findings or inconsistent results from well-designed, executed, and analyzed experiments.