Casting Light on Dependency Structures in Ensemble Forecasts With the 2-D Rank Histogram

A forecast is reliable if it is statistically indistinguishable from the observation in a distributional sense. In probabilistic forecasting, reliability is a necessary (but not sufficient) condition for optimal decision-making. In ensemble forecasting, reliability is the sign of a well-designed system. Tools for assessing reliability in the univariate case exist and have proved to be popular. One well-known example of a tool for ensemble forecasts is the rank histogram. Although univariate probabilistic forecasts are historically the most commonly used, multivariate forecasting is fundamental when multiple variables that influence each other play a role in a decision-making process. The simplest of the multivariate cases is the bivariate one, where only two interdependent variables are forecast. Here, we discuss how assessing the reliability of bivariate ensemble forecasts can be performed using generalisations of univariate diagnostic tools. We introduce the 2-D rank histogram, a simple and non-restrictive generalisation of the univariate rank histogram. A summary statistic of the ensemble reliability in the bivariate space is also suggested together with a strategy to disentangle marginal and dependency contributions. The interpretation of 2-D rank histograms is illustrated with synthetic data and ECMWF ensemble forecasts. Toy-model experiments are used to help associate histogram patterns with typical reliability misspecifications in a fully controlled environment, while an application to the ECMWF ensemble shows how reliability issues can be diagnosed with this versatile tool.