Repeated Combinatorial Test Design -- Unleashing the Potential in Multiple Testing Iterations

Abstract

Test design is the process of planning and designing the tests to be performedon a software system. The time scale at which organizations perform and modifytheir test design is typically orders of magnitude larger than that ofdevelopment, especially in modern agile development processes. As a result, often the carefully designed and optimized test suites end up beingrepeatedly executed with no variability between iterations, thus wasting crucialtime and resources. In this work we propose a repeated test planningprocess based on Combinatorial Test Design (CTD), where test iterations areplanned and executed while taking into account the previously executediterations. While each iteration satisfies the same test requirements as before, we leverage the degrees of freedom in test planning in order to reach fullcoverage of higher levels of requirements over the course of iterations. Wesuggest algorithms for doing so efficiently, and evaluate our approach over aset of models collected from different sources in literature. The evaluation demonstrates significant improvement in coverage rate of higher levels, comparedto the naïve approaches.