A cautionary tale of side-by-side evaluations while developing emotional expression for intelligent virtual agents

When designing interactive scenarios that depend on emotion expression, it is imperative to consider the levels of recognition associated with said expressions, to ascertain whether or not an acceptable degree of emotional communication has been achieved. In this work, two experiments were conducted with that aim, one asking participants to compare two different versions of an application side-by-side when conveying a specific emotion, and another asking the participants to recognize the emotion being expressed in each version. We found that, for some emotions, the approach rated higher in terms of emotion expression during the side-by-side comparison would not translate to the approach with a higher emotion recognition in the second experiment. Although this discrepancy is generally consistent with what happens with emotion recognition in humans, it is noteworthy that some higher-rated choices ended up not being as effective in the expression of emotion. We discuss how these discrepancies might have originated from forced-choice and feature dominance, and why context should be taken into account when designing experiments.