A methodology for evaluating predictive metrics

For over thirty years, software engineers have been interested in the ability to accurately measure characteristics of software and its production which could lead to improvements in both. In that time, a large number of metrics have been proposed, some with attempts at empirical validation of their effectiveness. Unfortunately, many if not most of these laudable efforts at empirical validation have foundered on a lack of knowledge about the appropriate methods to use. For example, a central goal in software metrics is the prediction of software characteristics based on other metrics of the software or its production process. This prediction problem is a quintessentially statistical one, but the lack of statistical training in the typical crowded engineering curriculum leaves most engineers uncertain about how to proceed. The result has been many well-intentioned but poorly executed empirical studies. This paper addresses this problem by providing a simple methodology for the predictive evaluation of metrics.