Improved Combination of Multiple Retrieval Systems Using a Dynamic Combinatorial Fusion Algorithm

Abstract

A combination of multiple retrieval systems can outperform its individual component systems, but it remains a challenging problem to predict whether two systems can be beneficially combined and, if so, the optimal means by which they should be merged. The performance of combined systems is affected by many factors, including the performance of individual systems, the diversity between a pair of systems, and the method for combination. In this paper, we undertake the study of these issues using combinatorial fusion algorithm (CFA) utilizing the rank-score characteristic (RSC) function and the notion of a weighted cognitive diversity. Using the selected eight TREC datasets, we demonstrated that: (a) the combination of two retrieval systems performs better than each individual system only when the individual systems have relatively good performance and they are diverse, (b) a dynamic combination method, using rank vs. score combination based on cognitive diversity which does not display a tight correlation with other statistical diversity measures, can improve the performance of the combined system, even when performance of each individual system is not known or in the context of an unsupervised learning environment. Within the TREC datasets, the proposed dynamic approach offers a potential for substantial improvement with no significant risk. Our results provide a new paradigm of dynamic fusion to the study of the combination of multiple retrieval systems.