Estimating the Reliability of the Retrieval Systems Rankings

Information retrieval evaluation based on the pooling method inherently poses biasness towards systems that contributed to the pool of judged documents. This may distort the results about the relative effectiveness of different retrieval strategies, or rather, the retrieval systems and thus result in unreliable system rankings. The purpose of this study is to suggest a technique to estimate the reliability of the retrieval system effectiveness rank in a list of ranked systems based on its performance in previous experiments. This can be also defined as the strength of rank for the individual retrieval system. By doing so, we will be able to predict the performance of each system in future experiments. To validate the proposed rank strength estimation method, an alternative systems ranking method is proposed to generate a new list of systems rankings which is used together with the proposed rank strength estimation method. The experimentation shows that the correlation coefficients remain above 0.8 across different number of experiments which means the new systems ranking is highly correlated with the gold standard. It suggests that the rank reliability estimation methods have effectively predicted the strength of the system ranks. And also, the results from both TREC 2004 and TREC 8 show the similar outcome which further confirms the effectiveness of the proposed rank reliability estimation method.