Robust recalibration of aggregate probability forecasts using meta-beliefs

Abstract

Previous work suggests that aggregate probabilistic forecasts on a binary event are often conservative. Extremizing transformations that adjust the aggregate forecast away from the uninformed prior of 0.5 can improve calibration in many settings. However, such transformations may be problematic in decision problems where forecasters share a biased prior. In these problems, extremizing transformations can introduce further miscalibration. We develop a two-step algorithm where we first estimate the prior using each forecaster’s belief about the average forecast of others. We then transform away from this estimated prior in each forecasting problem. Our algorithm works in single-question forecasting problems and does not require past data. Evidence from experimental prediction tasks suggests that the resulting average probability forecast is robust to biased priors and improves calibration.