Ranking Rich Mobile Verticals based on Clicks and Abandonment

We consider the problem of ranking rich verticals, which we call "cards," for a given mobile search query. Examples of card types include "SHOP" (showing access and contact information of a shop), "WEATHER" (showing a weather forecast for a particular location), and "TV" (showing information about a TV programme). These cards can be highly visual and/or concise, and may often satisfy the user's information need without making her click on them. While this "good abandonment" of the search engine result page is ideal especially for mobile environments where the interaction between the user and the search engine should be minimal, it poses a challenge for search engine companies whose ranking algorithms rely heavily on click data. In order to provide the right card types to the user for a given query, we propose a graph-based approach which extends a click-based automatic relevance estimation algorithm of Agrawal et al., by incorporating an abandonment-based preference rule. Using a real mobile query log from a commercial search engine, we constructed a data set containing 2,472 pairwise card type preferences covering 992 distinct queries, by hiring three independent assessors. Our proposed method outperforms a click-only baseline by 53-68% in terms of card type preference accuracy. The improvement is also statistically highly significant, with p ≈ 0.0000 according to the paired randomisation test.