Human recoverability index: A TraceLab experiment

Abstract

It has been generally accepted that not all trace links in a given requirements traceability matrix are equal - both human analysts and automated methods are good at spotting some links, but have blind spots for some other. One way to choose automated techniques for inclusion in assisted tracing processes (i.e., the tracing processes that combine the expertise of a human analyst and special-purpose tracing software) is to select the techniques that tend to discover more links that are hard for human analysts to observe and establish on their own. This paper proposes a new measure of performance of a tracing method: human recoverability index-based recall. In the presence of knowledge about the difficulty of link recovery by human analysts, this measure rewards methods that are able to recover such links over methods that tend to recover the same links as the human analysts. We describe a TraceLab experiment we designed to evaluate automated trace recovery methods based on this measure and provide a case study of the use of this experiment to profile and evaluate different automated tracing techniques.