Evaluating the Effectiveness of Regression Test Suites for Extract Method Validation

Refactoring edits aim to improve structural aspects of a system without changing its external behavior. However, while trying to perform a safe edit, a developer might introduce refactoring faults. To avoid refactoring faults, developers often use test suites to validate refactoring edits. However, depending on the quality of a test suite, its verdict may be misleading. In this work, we first present an empirical study that investigates the effectiveness of test suites (manually created and generated) for validating Extract Method refactoring faults. We found that manual suites detected 61,9% the injected faults, while generated suites detected only 46,7% (Randoop) and 55,8% (Evosuite). Then, we propose a new approach for evaluating the quality of a test suite for detecting refactoring faults. This approach is implemented by our prototype tool that focuses on two types of Extract Method faults. We demonstrate its applicability in a second empirical study that measured the quality of test suites from three different open-source projects.