Benchmarking COTS projects using data envelopment analysis

In Ernst & Young and Andersen Consulting, two of the "big five", there is a continuous search for better methods to measure and compare project performance of multi-dimensional COTS software projects. We propose using Data Envelopment Analysis (DEA) with a Variable Returns to Scale (VRS) model. First, we discuss and illustrate this method by analyzing Albrecht-Gaffney's two-dimensional dataset. Next, we review previous empirical studies using DEA showing that several studies have used DEA where simpler methods could have been used. Finally, we apply DEA to a multi-dimensional dataset of 30 industrial COTS software projects extracted from a benchmarking database in Andersen Consulting. Our main conclusion is that DEA is an applicable method, albeit not without shortcomings, for comparing the productivity of COTS software projects, and that it, therefore, merits further research. However, for two-dimensional datasets this method is unnecessary complex, and there exists other, simpler alternatives. Also, the results support our assumption of increasing as well as decreasing returns to scale for this dataset. Thus, the VRS model provides more reasonable and fair comparisons of project performance than a Constant Returns to Scale (CRS) model. Finally, this study suggests that DEA used together with methods for hypothesis testing may be a useful technique for assessing the effect of alleged process improvements.