Online metrics for web search relevance

Information Retrieval has a long tradition of being metrics driven. Ranking algorithms are assessed with respect to some utility measure that reflects the likelihood of satisfying an information need. Traditionally these metrics are based on offline judgments. This is very flexible since judgments can be made for any desired output. However, judgments are no better than judgment guidelines and are at some distance from the actual user experience. Modern Web Search engines enjoy an additional resource; existing web search traffic and its attendant wealth of user engagement data. Primarily this signal consists of logged queries and user actions, including clicks and reformulations. I will discuss how this data can be used to derive Web Search quality metrics that have very different properties than traditional offline metrics.