History of a Large Test Automation Project Using Selenium

Abstract

In 2007 I started work as a tester for a company called Socialtext. When I joined the company there was already a Selenium-based test framework in place, but there were only a couple of automated test cases created; we had about 400 test steps, or individual assertions about the behavior of the application. When I left Socialtext two years later, we had just surpassed 10,000 test steps in the main set of regression tests. We also had browser-specific test sets in place, an automated test case for visually checking the application, and a Continuous-Integration-like script that ran all day and all night against the latest version of the code. At about 4000 test steps, regression bugs released to production dropped essentially to zero. The other 6000 test steps covered ongoing new features in the project, and more robust testing of the older application functions. This report discusses how I helped grow this system, and the things we learned along the way that helped it be such a successful ongoing project. The report covers initial conditions and test design; discusses issues in application feature coverage; how and when to grow the system quickly; a couple of test design smells that caused us problems along the way; how we treat Continuous Integration in a system like this; and how we coped when significant parts of the User Interface were completely re-engineered.