Balancing Aspects in Retrieved Search Results

Many queries contain explicit aspects which must be balanced in any retrieved result in order to meet a user's information need: if aspects of the query are missing or disproportionately represented in documents, the results will be of lower quality than desired. This balancing thus needs to occur both within the retrieved documents individually and across the entire set. We introduce the concept of query-aspect balance and describe a new evaluation measure, β-NDCG, that allows the evaluation of query-aspect balance on multivalued query-aspect judgments. We apply β-NDCG to a small test collection and explore its utility. We show that β-NDCG-NDCG captures problems of query aspect balance within and across documents in the ranked list.