Retrofitting an acceptance test framework for clarity

Abstract

An XP customer needs to write and check acceptance tests. However, the format for defining the tests needs to be clear. Many acceptance test approaches use arcane formats which do not promote clarity for the customer, due to a conflict of interest between the complexities of automation and the needs of the customer. We discuss the evolution of acceptance tests to improve their clarity for the customer. Sat is an acceptance test system for testing socket-based servers with multiple clients. The first version used an XML file to define the tests in a test suite. Any errors detected were written to a text log. There were two problems with this first version. The XML format made it difficult to read and edit the tests. When an error was given, it was not easy to identify the place in the test where the problem occurred. Sat was altered to make use of Fit, a testing framework that uses HTML tables for defining tests and reporting any errors. We found the new version considerably easier to use. The tabular form makes it much simpler to read and alter the tests. Any errors are reported in a copy of the tables, in the place where they occur. We have also found it convenient to include information about the tests in the HTML, providing a form of "literate testing".