Competing Forecast Verification: Using the Power-Divergence Statistic for Testing the Frequency of “Better”

Abstract

When testing hypotheses about which of two competing models is better, say A and B, the difference is often not significant. An alternative, complementary approach, is to measure how often model A is better than model B regardless of how slight or large the difference. The hypothesis concerns whether or not the percentage of time that model A is better than model B is larger than 50%. One generalized test statistic that can be used is the power-divergence test, which encompasses many familiar goodness-of-fit test statistics, such as the loglikelihood-ratio and Pearson X2 tests. Theoretical results justify using the distribution for the entire family of test statistics, where k is the number of categories. However, these results assume that the underlying data are independent and identically distributed; which is often violated. Empirical results demonstrate that the reduction to two categories (i.e., model A is better than model B v. model B is better than A) results in a test that is reasonably robust to even severe departures from temporal independence, as well as contemporaneous correlation. The test is demonstrated on two different example verification sets: 6-h forecasts of eddy dissipation rate (m2/3s−1) from two versions of the Graphical Turbulence Guidence model and for 12-hour forecasts of 2-m temperature (°C) and 10-m wind speed (ms−1) from two versions of the High-Resolution Rapid Refresh model. The novelty of this paper is in demonstrating the utility of the power-divergence statistic in the face of temporally dependent data, as well as the emphasis on testing for the “frequency-of-better” alongside more traditional measures.