Model agnostic interpretability of rankers via intent modelling

Abstract

A key problem in information retrieval is understanding the latent intention of a user's under-specified query. Retrieval models that are able to correctly uncover the query intent often perform well on the document ranking task. In this paper we study the problem of interpretability for text based ranking models by trying to unearth the query intent as understood by complex retrieval models. We propose a model-agnostic approach that attempts to locally approximate a complex ranker by using a simple ranking model in the term space. Given a query and a blackbox ranking model, we propose an approach that systematically exploits preference pairs extracted from the target ranking and document perturbations to identify a set of intent terms and a simple term based ranker that can faithfully and accurately mimic the complex blackbox ranker in that locality. Our results indicate that we can indeed interpret more complex models with high fidelity. We also present a case study on how our approach can be used to interpret recently proposed neural rankers.