Measuring probabilistic coherence to identify superior forecasters

Abstract

Forecasts, or subjective probability assessments of uncertain events, are characterized by two qualities: coherence, the degree to which the judgments are internally consistent, and correspondence, the extent to which judgments are accurate. Recent evidence suggests that more coherent forecasts tend to be more accurate. However, currently, there is no good stand-alone measure of probabilistic coherence. We developed and validated the Coherence Forecasting Scale (CFS). This questionnaire assesses how well people understand and apply probabilistic reasoning rules such as relations between joint and disjoint probabilities, probability complementarity, stochastic dominance, and monotonicity. In three incentivized forecasting tournaments, including one from an online public forecasting platform, judges who scored higher on the CFS were also more accurate. Notably, across all tournaments, the CFS dominates all administered individual difference and demographic measures in explanatory power predicting judgment accuracy, providing empirical evidence that coherence and accuracy are strongly linked.