Preference or Confidence? Untangling Heterogeneity in Paired Comparisons.

Abstract

Paired comparison models analyze data from experiments in which items are compared two at a time. Each item is often associated with a strength parameter, and parameter differences determine choice probabilities. The parameters carry two kinds of information. Their order encodes ordinal preferences, and their magnitude reflects the confidence or consistency with which those preferences are expressed. Standard paired comparison models often assume that a single set of strength parameters characterizes all decision-makers. One way to relax this assumption is to fit a mixture model with a fixed number of groups, each with its own parameter set. However, if individual variation in confidence is not modeled, estimation may allocate the limited number of groups to capture magnitude differences rather than differences in order. This article presents a framework that integrates random effects into mixture models, allowing population subgroups and individual-level confidence variation to be estimated simultaneously. By absorbing within-group confidence variation, the available groups are freed to capture meaningful differences in preference order. Simulation studies show improved estimation and interpretability and demonstrate that covariate-based partitioning cannot replace direct modeling of intra-individual variation. Model performance is illustrated using data on juror perceptions of forensic evidence reports.