Large-scale test automation in the cloud (invited industrial talk)

Abstract

Software development at Google is big and fast. The code base receives 20+ code changes per minute and 50% of the files change every month! Each product is developed and released from head relying on automated tests verifying the product behavior. Release frequency varies from multiple times per day to once every few weeks, depending on the product team. With such a huge, fast-moving codebase, it is possible for teams to get stuck spending a lot of time just keeping their build green. A continuous integration system should help by providing the exact change at which a test started failing, instead of a range of suspect changes or doing a lengthy binary-search for the offending change. We have built a system that uses dependency analysis to determine all the tests a change transitively affects and then runs only those tests for every change. The system is built on top of Google's cloud computing infrastructure enabling many builds to be executed concurrently, allowing the system to run affected tests as soon as a change is submitted. The use of smart tools and cloud computing infrastructure in the continuous integration system enables quick, effective feedback to development teams.