Judgment Pool Effects Caused by Query Variations

Batch-mode retrieval evaluation relies on suitable relevance judgments being available. Here we explore the implications on pool size of adopting a "query variations" approach to collection construction. Using the resources provided as part of the UQV100 collection [Bailey et al., SIGIR 2016] and a total of five different systems, we show that pool size is as much affected by the number of query variations involved as it is by the number of contributing systems, and that systems and users are independent effects. That is, if both system and query variation are to be accommodated in retrieval experimentation, the cost of performing the required judgments compounds.