Detecting cognitive bias in a relevance assessment task using an eye tracker

Abstract

Cognitive biases, such as the bandwagon effect, occur when a participant places a disproportionate emphasis on external information when making decisions under uncertainty. These effects are challenging for humans to overcome - even when they are explicitly made aware of their own biases. One challenge for researchers is to detect if the information is used in decision making and to what degree. One can gain a better understanding of how this external information is used in decision making using an eye tracker. In this paper, we evaluate cognitive biases in the context of assessing the binary relevance of a set of documents in response to a given information need. We show that these cognitive biases can be observed by examining gaze time in Areas of Interest (AOI) that contain this pertinent external information.