Using a concept map to evaluate exploratory search

Users often conduct exploratory searches on the Web to solve complex problems, but how for evaluating the effectiveness of a search, especially in a user-centered approach, remains an open question. We propose a user-centered method to evaluate the effectiveness of an exploratory search by focusing on the change in the users' mental representations of a topic during their exploratory search on the Web. This was done by comparing the concept maps depicted before and after each users searching. Thirty-five undergraduate students recruited from various departments and universities participated in our experiment. The participants were instructed to search for and gather Web pages for the task of writing a term paper on two given topics, politics and media, while working in either of two scenarios: "Selective scenario" (identifying only ten Web pages as important) and "As-Many-As scenario" (collecting as many Web pages as possible). The participants were divided into two groups: one group searched for both topics in the Selective scenario, and the other searched for them in the As-Many-As scenario. They composed their concept maps before and after searching. We defined the following measures to show the differences between the maps before and after a search to analyze the concept maps made by the participants: common, new, and lost map components like nodes, links, and link labels. We used these measures to compare the results from different topics, scenarios, and browser types. The results showed that the concept maps greatly changed after searching and that there were more common nodes and link labels in the Selective scenario than in the As-Many-As scenario. A comparative analysis of the concept maps between the pre- and post-search maps indicated that the users significantly changed their knowledge structure of a topic by completing the exploratory search task.