Time well spent

Abstract

Time-biased gain provides a general framework for predicting user performance on information retrieval systems, capturing the impact of the user's interaction with the system's interface. Our prior work investigated an instantiation of time-biased gain aimed at traditional search interfaces utilizing clickable result summaries, with gain realized from the recognition of relevant documents. In this paper, we examine additional properties of time-biased gain, demonstrating how it generalizes effectiveness measures from across the field of information retrieval. We explore a new instantiation of time-biased gain, applicable to systems where the user judges the quality of their experience by the amount of time well spent. Rather than the single number produced by traditional effectiveness measures, time-biased gain models user variability and produces a distribution of gain on a per-query basis. With this distribution, we can observe performance differences at the user level. We apply bootstrap sampling to estimate confidence intervals across multiple queries.