A Kappa Measurement of Query Consistency and Its Application

Abstract

To measure the diversity of the user interests over the same query, this paper applies kappa coefficient as the indicator of the consistency of users’ clicks for a given query. It compares three different settings of the Kappa parameters and shows the Kappa formula can be well adapted to the Query log analysis. Based on the further analysis of the Kappa results over Sogou Query LOG, it is revealed that the user interests over the same query are rather diversified, which indicates that the personalized information retrieval is a promising solution to improve the current information retrieval performance.