Incremental refinement of relevance rankings: Balancing relevance depth and scope

Abstract

Delivering both relevant and topically diverse results is a key challenge in information retrieval (IR). This study introduces a hybrid method that incrementally refines rankings by combining probabilistic topic modeling (latent dirichlet allocation [LDA]) with citation-based pennant retrieval grounded in Relevance Theory (RT), optimizing for both relevance and diversity. A unified scoring function fuses topic and citation cues, while a tunable weight parameter makes the relevance–diversity trade-off explicit and transparent. Applying this RT-guided approach to the iSearch corpus (435,000 physics papers, 3.7 million citations) across 65 queries, we evaluate relevance and diversity via degree centrality, arXiv subject categories, and maximal marginal relevance (MMR). Unlike prior methods that apply LDA or pennant retrieval separately—or depend on post-hoc diversification such as MMR—this approach integrates them directly and efficiently: LDA runs on local multi-core CPUs, and pennant retrieval operates using sparse co-citation counts. The combined technique retrieves both central and peripheral works, producing a boundary-spanning effect that broadens conceptual coverage while improving normalized discounted cumulative gain (NDCG) performance. Interactive visualizations via LuminaCite depict how researchers can fine-tune results in real time to reveal latent or interdisciplinary connections. Sensitivity and robustness tests indicate that the proposed model is robust to hyperparameter variation, first-stage retriever choice, and query reformulation.