Incorrect inference from size-selectivity studies due to widespread misuse of bootstrap confidence bands

Abstract

The bootstrap is now widely used in size-selectivity experiments to quantify the uncertainty in estimated absolute or relative selectivity curves. A two-level hierarchical bootstrap is commonly used, with deployments being resampled at the first level followed by resampling of individual fish at the second level. It is standard practice to construct plots of the estimated curves with an overlay of the bootstrap confidence intervals for each length, thereby forming a bootstrap confidence band. These confidence bands are routinely used to perform statistical tests, most often to test the null hypothesis that two gears have the same length-dependent selectivity and/or fishing efficiency. Using computer simulation, it is shown here that the actual Type 1 error (non-coverage) rate of the bootstrap confidence intervals can vary considerably depending on the experimental scenario. The bootstrap confidence band test was found to inflate the Type 1 error rate of the confidence intervals by a factor of about five due to the use of multiple confidence intervals in the construction of a confidence band. Moreover, the bootstrap confidence band test can have very low statistical power, as demonstrated with an example in which a permutation test has 97% power to detect a difference between two fishing gears, whereas the bootstrap confidence band test has power of just 68%. The permutation test is recommended for testing differences between the size selectivity of gears and its ease of implementation is demonstrated. The misuse of bootstrap confidence bands has recently propagated to other areas of fishing-gear selectivity research, including assessment of differences in species selectivity between gears, and this is another situation where permutation tests would be straightforward to implement.