Learning to be confident: How agents learn confidence based on prediction errors

Abstract

Decision confidence should normatively reflect the posterior probability of making a correct choice, conditional on relevant information. However, how individuals learn to calibrate their sense of confidence to that probability remains unknown. The standard approach to estimate any quantity is to use trial-by trial samples of that quantity to train a function approximator (such as a neural network) based on the prediction errors (quantity minus prediction of the quantity). We tested whether humans learn about confidence using this principle in a perceptual decision-making experiment where participants repeatedly alternated between two manipulated feedback regimes (negative vs positive) every few blocks of trials. As anticipated, confidence ratings tracked feedback, with confidence gradually increasing when participants received overall positive feedback (and thus positive prediction errors), and decreasing when receiving negative feedback (and thus negative prediction errors). These feedback-induced dynamic changes were specific to confidence, as objective performance was unaffected by the manipulation. We propose a single-layer neural network model for confidence which updates the computation of confidence based on trial-level prediction errors, and demonstrate that it better fits the behavioral data compared to a purely valence-based model. Taken together, these results show that the computation of confidence is dynamic: humans constantly update how they compute confidence based on prediction errors (feedback minus prediction), in a statistically efficient manner.