"Learning Relevance" as a Service for Improving Search Results in Technical Discussion Forums

Search results in technical forums are typically keyword based. The relevance of a link is usually gauged by closest content match. However, it has been shown in literature that users' click behavior is an integral part of deciding the relevance of a search result. Moreover, it is not just the number of clicks that matter, but time spent on a clicked link, order in which the links were clicked etc. also play an important role in the relevance decision. In this paper, we have developed a service that analyzes the click logs of searches performed in the technical forums and learns the new relevance scores for the search results with respect to a query. The computation model for relevance is an optimization problem, the constraints for which have been designed based on real user behavior study. We ingested StackOverflow data for few domains and designed a QA style search to carry out the study. We have developed heuristics to solve the optimization problem and have validated the relevance model using user behavior simulations. The relevance model is shown to yield efficient, robust and effective rank order using DCG (discounted cumulative gains) and stability metrics.