Resource/Schedule/Content Test Planning Model

It is necessary to balance the contributions from four primary software testing 'dimensions,' in the integration branch test cycle for waterfall and hybrid waterfall/agile projects, to achieve best-in-class customer experience: 1) Sufficient testing resources (engineers) are needed to ensure that adequate testing is accomplished, 2) sufficient bug fixing resources (engineers) are needed to ensure that the newly uncovered bugs are properly fixed, 3) adequate testing and bug fixing time/schedule are needed to ensure that there is sufficient time to run the test plans completely, to do enough regression testing, and to accommodate dead periods when test stoppers are encountered, and 4) new feature content must not be so high that the testing and fixing teams cannot complete their tasks and produce code that is sufficiently reliable. These four dimensions are key parameters in any testing planning exercise, and often, during the testing cycle itself, one or more of these parameters must be adjusted to satisfy the cost/schedule/reliability goals laid out at the project start. The work described in this paper attempts to construct a generalized model that quantifies the contributions from these dimensions, and enables the practitioner to construct what-if scenarios to dynamically estimate the customers' software reliability experience in their production networks. This Resource/Schedule/ Content Model (RSCM), addressing resources, schedule, and feature content, is based on another useful model, the Universal Release Criteria Model (URC), that gauges the effectiveness of integration-branch testing and fixing, and estimates the software reliability field results. The customer experience metric that is predicted by RSCM and URC is SWDPMH -- software incidents (bug encounters by all customers) per million hours of total fleet usage per month, Cisco's primary measure of reliability in the field.