{"title":"使用多元宇宙方法评估政府应对流行病影响的研究的潜在偏差和有效性分析","authors":"Jeremy D. Goldhaber-Fiebert","doi":"arxiv-2409.06930","DOIUrl":null,"url":null,"abstract":"We analyze the methodological approach and validity of interpretation of\nusing national-level time-series regression analyses relating epidemic outcomes\nto policies that estimate many models involving permutations of analytic\nchoices (i.e., a \"multiverse\" approach). Specifically, we evaluate the possible\nbiases and pitfalls of interpretation of a multiverse approach to the context\nof government responses to epidemics using the example of COVID-19 and a\nrecently published peer-reviewed paper by Bendavid and Patel (2024) along with\nthe subsequent commentary that the two authors published discussing and\ninterpreting the implications of their work. While we identify multiple\npotential errors and sources of biases in how the specific analyses were\nundertaken that are also relevant for other studies employing similar\napproaches, our most important finding involves constructing a counterexample\nshowing that causal model specification-agnostic multiverse analyses can be\nincorrectly used to suggest that no consistent effect can be discovered in data\nespecially in cases where most specifications estimated with the data are far\nfrom causally valid. Finally, we suggest an alternative approach involving\nhypothesis-drive specifications that explicitly account for infectiousness\nacross jurisdictions in the analysis as well as the interconnected ways that\npolicies and behavioral responses may evolve within and across these\njurisdictions.","PeriodicalId":501219,"journal":{"name":"arXiv - QuanBio - Other Quantitative Biology","volume":"48 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-09-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Analysis of Potential Biases and Validity of Studies Using Multiverse Approaches to Assess the Impacts of Government Responses to Epidemics\",\"authors\":\"Jeremy D. Goldhaber-Fiebert\",\"doi\":\"arxiv-2409.06930\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"We analyze the methodological approach and validity of interpretation of\\nusing national-level time-series regression analyses relating epidemic outcomes\\nto policies that estimate many models involving permutations of analytic\\nchoices (i.e., a \\\"multiverse\\\" approach). Specifically, we evaluate the possible\\nbiases and pitfalls of interpretation of a multiverse approach to the context\\nof government responses to epidemics using the example of COVID-19 and a\\nrecently published peer-reviewed paper by Bendavid and Patel (2024) along with\\nthe subsequent commentary that the two authors published discussing and\\ninterpreting the implications of their work. While we identify multiple\\npotential errors and sources of biases in how the specific analyses were\\nundertaken that are also relevant for other studies employing similar\\napproaches, our most important finding involves constructing a counterexample\\nshowing that causal model specification-agnostic multiverse analyses can be\\nincorrectly used to suggest that no consistent effect can be discovered in data\\nespecially in cases where most specifications estimated with the data are far\\nfrom causally valid. Finally, we suggest an alternative approach involving\\nhypothesis-drive specifications that explicitly account for infectiousness\\nacross jurisdictions in the analysis as well as the interconnected ways that\\npolicies and behavioral responses may evolve within and across these\\njurisdictions.\",\"PeriodicalId\":501219,\"journal\":{\"name\":\"arXiv - QuanBio - Other Quantitative Biology\",\"volume\":\"48 1\",\"pages\":\"\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-09-11\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - QuanBio - Other Quantitative Biology\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2409.06930\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - QuanBio - Other Quantitative Biology","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.06930","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
Analysis of Potential Biases and Validity of Studies Using Multiverse Approaches to Assess the Impacts of Government Responses to Epidemics
We analyze the methodological approach and validity of interpretation of
using national-level time-series regression analyses relating epidemic outcomes
to policies that estimate many models involving permutations of analytic
choices (i.e., a "multiverse" approach). Specifically, we evaluate the possible
biases and pitfalls of interpretation of a multiverse approach to the context
of government responses to epidemics using the example of COVID-19 and a
recently published peer-reviewed paper by Bendavid and Patel (2024) along with
the subsequent commentary that the two authors published discussing and
interpreting the implications of their work. While we identify multiple
potential errors and sources of biases in how the specific analyses were
undertaken that are also relevant for other studies employing similar
approaches, our most important finding involves constructing a counterexample
showing that causal model specification-agnostic multiverse analyses can be
incorrectly used to suggest that no consistent effect can be discovered in data
especially in cases where most specifications estimated with the data are far
from causally valid. Finally, we suggest an alternative approach involving
hypothesis-drive specifications that explicitly account for infectiousness
across jurisdictions in the analysis as well as the interconnected ways that
policies and behavioral responses may evolve within and across these
jurisdictions.