Introduction to the Special Issue on Reproducibility in Information Retrieval

Information Retrieval (IR) is a discipline that has been strongly rooted in experimentation since its inception. Experimental evaluation has always been a strong driver for IR research and innovation, and these activities have been shaped by large-scale evaluation campaigns such as Text REtrieval Conference (TREC) in the U.S., Conference and Labs of the Evaluation Forum (CLEF) in Europe, NII Testbeds and Community for Information access Research (NTCIR) in Japan and Asia, and Forum for Information Retrieval Evaluation (FIRE) in India. IR systems are getting increasingly complex. They need to cross language and media barriers; they span from unstructured, via semi-structured, to highly structured data; and they are faced with diverse, complex, and frequently underspecified (ambiguously specified) information needs, search tasks, and societal challenges. As a consequence, evaluation and experimentation, which has remained a fundamental element, has in turn become increasingly sophisticated and challenging.