SCAMP: a tool for conducting interactive information retrieval experiments

Abstract

Conducting Interactive Information Retrieval (IIR) research is often seen as an arduous and tedious process with a high barrier to entry. This high barrier is due to the overheads in developing and setting up even a simple lab-based IIR experiment. SCAMP (Search ConfigurAtor for experiMenting with PuppyIR) is a web-based tool that we have developed which enables researchers to configure standard IIR experiments. SCAMP provides the infrastructure that handles the major processes within the experimental flow (such as Participant Registration, Consent, Surveys, and the logging and tracking of tasks and participants through the experiment). Consequently, SCAMP reduces the time required to create an experiment and is ideal for undergraduate and masters students who would like to conduct an IIR experiment without extensive development. Furthermore, the tool is extensible. Other features can be easily added to SCAMP to customise the experiments. To evaluate SCAMP, we performed a within-subjects experiment where 48 participants used a web search engine with different search aids (query suggestion, spell correction, etc) to complete various web search tasks - to increase the difficulty of the search tasks certain query terms were banned. We use this evaluation to showcase the different features of the SCAMP system and report on how participants perform under difficult querying conditions.