Measuring the Effectiveness of Selective Search Index Partitions without Supervision

Selective search architectures partition a document collection into topic-oriented index shards, usually using algorithms that have random components. Different mappings of documents into index shards (shard maps) produce different search accuracy and consistency, however identifying which shard maps will deliver the highest average effectiveness is an open problem. This paper presents a new metric, Area Under Recall Curve (AUReC), to evaluate and compare shard maps. AUReC is the first such metric that is independent of resource selection and shard cut-off estimation. It does not require an end-to-end evaluation or manual gold-standard judgements. Experiments show that its predictions are highly-correlated with evaluating end-to-end systems of various configurations, while being easier to implement and computationally inexpensive.