Reusability assessment of test collections with multi-levels of judgments

Constructing good test collection is an expensive and time-consuming process. Traditionally, test collections contain binary judgments. In recent years, however, there has been increasingly interest in test collections with Multi-levels judgments and of certain qualities. Such collections are even more expensive to construct. Therefore, ability to reuse test collections can not only save construction costs, but also boosts our confidence in their quality. This paper proposes a method for assessing reusability of a test collection with multi-level judgments. The proposed method can help IR researchers to determine whether an existing test collection with a set of multi-level judgments is suitable for evaluating a new IR system or not. Results of our experiments (on MAHAK test collection) suggest that this method can help assessing reusability of a test collection.