Reevaluating the reliability of common multiple comparison tests

Abstract

Many comparison tests are available to determine treatment differences. The validity of these tests is commonly assessed using the Type I error rate. Type I error is obtaining a false positive result. It is known that Fisher's Least Significant Difference (LSD) and Duncan tests have high Type I error rates because even a single false positive is sufficient to constitute a Type I error. For multiple comparison tests, the number of correct decisions (true positives and true negatives) is more important than the Type I error rate. Therefore, in this study, specificity and sensitivity were considered alongside the Type I error rate. Specificity refers to the true negative rate, while sensitivity refers to the true positive rate. A Monte Carlo simulation showed that the LSD and Duncan tests had relatively high Type I error rates; however, when specificity was considered, the LSD and Duncan tests correctly predicted statistically similar groups (true negative) with a mean of 97.00%, while other tests achieved 99.00%. Regarding sensitivity, the LSD and Duncan tests correctly identified statistically different groups (true positive) with a mean of 15.00%, while other tests achieved 3.00%. The true negative rate of the other tests is 1.02 times (99.00/97.00) that of LSD and Duncan. In contrast, the true positive rate of LSD and Duncan is 5.00 times (15.00/3.00) that of the other tests. Therefore, considering both specificity and sensitivity, the LSD and Duncan tests were found to be superior to others. In conclusion, these tests were shown to be more reliable.