Evaluation of Retrieval Algorithms for Expertise Search

Evaluation of expertise search systems is a non-trivial task. While in a typical search engine the responses to user queries are documents, the search results for an expertise retrieval system are people. The relevancy scores indicate how knowledgeable they are on a given topic. Within an organisation, such a ranking of employees could potentially be difficult as well as controversial. We introduce an in-house capability search system built for an organisation with a diverse range of disciplines. We report on two attempts of evaluating six different ranking algorithms implemented for this system. Evaluating the system using relevance judgements produced in each of the two attempts leads to an understanding of how different methods of collecting judgements on people's expertise can lead to different effectiveness of algorithms.