Prevalence of continuous integration failures in industrial systems with hardware-in-the-loop testing

Abstract

Faults in the automated continuous integration (CI) process can seriously impact the development of industrial code. To reduce manual intervention in automated CI processes, we want to understand better the CI systems' failure distribution to improve efficiency, reliability, and maintainability. This paper investigates failures in CI in four large industrial projects. We gather 11 731 builds over six months, identifying 1 414 failing builds. We also identify the distribution of different types of build failures in each of the four CI projects. Our results show that compilation is the most significant individual cause of failure with 47 %, followed by testing at 36 %. The checkout step with associated checks also incurs a non-negligible portion of failures with 12 %. Furthermore, we identify 14 distinct types of failures in the testing step. We conclude that configuration problems are a significant issue, as pipeline scripting and dependency errors make up a large number of failures.