A Comparative Evaluation on the Quality of Manual and Automatic Test Case Generation Techniques for Scientific Software - a Case Study of a Python Project for Material Science Workflows

Writing software tests is essential to ensure a high quality of the software project under test. However, writing tests manually is time consuming and expensive. Especially in research fields of the natural sciences, scientists do not have a formal education in software engineering. Thus, automatic test case generation is particularly promising to help build good test suites. In this case study, we investigate the efficacy of automated test case generation approaches for the Python project Atomic Simulation Environment (ASE) used in the material sciences. We compare the branch and mutation coverages reached by both the automatic approaches, as well as a manually created test suite. Finally, we statistically evaluate the measured coverages by each approach against those reached by any of the other approaches. We find that while all evaluated approaches are able to improve upon the original test suite of ASE, none of the automated test case generation algorithms manage to come close to the coverages reached by the manually created test suite. We hypothesize this may be due to the fact that none of the employed test case generation approaches were developed to work on complex structured inputs. Thus, we conclude that more work may be needed if automated test case generation is used on software that requires this type of input.