Beyond Randomization: Design and Analysis of Discrete Choice Experiments in the Presence of Profile Order Effects Within Choice Sets

Abstract

Discrete Choice Experiments (DCEs) investigate the attributes that affect individual choices among different options and are widely applied across numerous fields. Past DCEs provide clear evidence that the presentation order of the profiles within a choice set can impact the respondents' choices. Ignoring such order effects can produce severely biased estimates, as we illustrate using a product packaging DCE performed for Procter & Gamble in Mexico. Currently, the most common approach to address profile order effects is to randomize the profile order. While this method is relatively easy to implement in online surveys, it can be considerably cumbersome in offline experimental settings. To address this, we suggest incorporating an order covariate in the model to measure the effect of profile order, and propose a Bayesian optimal Balanced Profile Order Design (BPOD) that accounts for this order effect. Our simulation experiments reveal that our Bayesian optimal BPOD achieves accurate parameter estimates comparable to those obtained through randomization in both the multinomial logit model and the panel mixed logit model. Beyond DCEs, this design strategy contributes to broader efforts in experimental design by providing a generalizable framework for addressing structural sources of bias in applied statistical research.