Towards incremental component compatibility testing

Abstract

Software components are increasingly assembled from other components. Each component may further depend on others, and each may have multiple active versions. The total number of configurations --- combinations of components and their versions --- deployed by end users can be very large. Component developers, therefore, spend considerable time and effort doing compatibility testing --- determining whether their components can be built correctly for all deployed configurations. In previous work we developed Rachet to support large-scale compatibility testing of components. In this paper, we describe and evaluate methods to enable Rachet to perform incremental compatibility testing. We describe algorithms to compute differences in component compatibilities between current and previous component builds, a formal test adequacy criterion based on covering the differences, and cache-aware configuration sampling and testing methods that attempt to reuse effort from previous testing sessions. We evaluate our approach using the 5-year evolution history of a scientific middleware component. Our results show significant performance improvements over Rachet's previous retest-all approach, making the process of compatibility testing practical for evolving components.