{"title":"Counternull Sets in Randomized Experiments","authors":"M.-A. C. Bind, D. B. Rubin","doi":"10.1080/00031305.2024.2432884","DOIUrl":null,"url":null,"abstract":"Consider a study whose primary results are “not statistically significant”. How often does it lead to the following published conclusion that “there is no effect of the treatment/exposure on the outcome”? We believe too often and that the requirement to report counternull values could help to avoid this! In statistical parlance, the null value of an estimand is a value that is distinguished in some way from other possible values, for example a value that indicates no difference between the general health status of those treated with a new drug versus a traditional drug. A counternull value is a nonnull value of that estimand that is supported by the same amount of evidence that supports the null value. Of course, such a definition depends critically on how “evidence” is defined. Here, we consider the context of a randomized experiment where evidence is summarized by the randomization-based <i>p</i>-value associated with a specified sharp null hypothesis. Consequently, a counternull value has the same <i>p</i>-value from the randomization test as does the null value; the counternull value is rarely unique, but rather comprises a <i>set</i> of values. We explore advantages to reporting a counternull set in addition to the <i>p</i>-value associated with a null value; a first advantage is pedagogical, in that reporting it avoids the mistake of implicitly accepting a not-rejected null hypothesis; a second advantage is that the effort to construct a counternull set can be scientifically helpful by encouraging thought about nonnull values of estimands. Two examples are used to illustrate these ideas.","PeriodicalId":50801,"journal":{"name":"American Statistician","volume":"1 1","pages":""},"PeriodicalIF":1.8000,"publicationDate":"2025-04-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"American Statistician","FirstCategoryId":"100","ListUrlMain":"https://doi.org/10.1080/00031305.2024.2432884","RegionNum":4,"RegionCategory":"数学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"STATISTICS & PROBABILITY","Score":null,"Total":0}
引用次数: 0
Abstract
Consider a study whose primary results are “not statistically significant”. How often does it lead to the following published conclusion that “there is no effect of the treatment/exposure on the outcome”? We believe too often and that the requirement to report counternull values could help to avoid this! In statistical parlance, the null value of an estimand is a value that is distinguished in some way from other possible values, for example a value that indicates no difference between the general health status of those treated with a new drug versus a traditional drug. A counternull value is a nonnull value of that estimand that is supported by the same amount of evidence that supports the null value. Of course, such a definition depends critically on how “evidence” is defined. Here, we consider the context of a randomized experiment where evidence is summarized by the randomization-based p-value associated with a specified sharp null hypothesis. Consequently, a counternull value has the same p-value from the randomization test as does the null value; the counternull value is rarely unique, but rather comprises a set of values. We explore advantages to reporting a counternull set in addition to the p-value associated with a null value; a first advantage is pedagogical, in that reporting it avoids the mistake of implicitly accepting a not-rejected null hypothesis; a second advantage is that the effort to construct a counternull set can be scientifically helpful by encouraging thought about nonnull values of estimands. Two examples are used to illustrate these ideas.
期刊介绍:
Are you looking for general-interest articles about current national and international statistical problems and programs; interesting and fun articles of a general nature about statistics and its applications; or the teaching of statistics? Then you are looking for The American Statistician (TAS), published quarterly by the American Statistical Association. TAS contains timely articles organized into the following sections: Statistical Practice, General, Teacher''s Corner, History Corner, Interdisciplinary, Statistical Computing and Graphics, Reviews of Books and Teaching Materials, and Letters to the Editor.