Predicting field experience of releases on specific platforms

Abstract

Since 2009, Software Defects Per Million Hours (SWDPMH) has been the primary customer experience metric used at Cisco, and is goaled on a yearly basis for about 100 product families. A key reason SWDPMH is considered to be of critical importance is that we see a high correlation between SWDPMH and Software Customer Satisfaction (SW CSAT) over a wide spectrum of products and feature releases. Therefore, it is important to try to anticipate SWDPMH for new releases before the software is released to customers, for several reasons: · Early warning that a major feature release is likely to experience substantial quality problems in the field may allow for remediation of the release during, or even prior to, function and system testing · Prediction of SWDPMH enables better planning for subsequent maintenance releases and rollout strategies · Calculating the tradeoffs between SWDPMH and feature volume can provide guidance concerning acceptable feature content, test effort, release cycle timing, and other key parameters affecting subsequent feature releases. Our efforts over the past year have been to enhance our ability to predict SWDPMH in the field. Toward this end, we have developed predictive models, tested the models with major feature releases for strategic products, and provided guidance to development, test, and release management teams on how to improve the chances of achieving best-in-class levels of SWDPMH. This work is ongoing, but several models are currently used in a production mode for five product families, with good results. We plan to achieve production capability with an additional several dozen product families over the next year.