Reliable testing: detecting state-polluting tests to prevent test dependency

Abstract

Writing reliable test suites for large object-oriented systems is complex and time consuming. One common cause of unreliable test suites are test dependencies that can cause tests to fail unexpectedly, not exposing bugs in the code under test but in the test code itself. Prior research has shown that the main reason for test dependencies is the ``pollution'' of state shared across tests. We propose a technique, called , for finding tests that pollute the shared state. In a nutshell, finds tests that modify some location on the heap shared across tests or on the file system; a subsequent test could fail if it assumes the shared location to have the initial value before the state was modified. To aid in inspecting the pollutions, provides an access path through the heap that leads to the polluted value or the name of the file that was modified. We implemented a prototype tool for Java and evaluated it on NumOfProjects projects, with a total of NumOfTests tests. Diaper reported PollutingTests , and our inspection found that NumOfTPsSpace of those are relevant pollutions that can easily affect other tests.