So much more than test cases – An industrial study on testing of software units and components

While software-developing organizations increasingly write test cases and continuously run them to find defects and ensure quality, the associated maintenance and execution costs increase over time. According to practitioners in the software industry, more test cases is not the solution, which calls for other methods and tools that can complement traditional test cases. Based on a cross-company workshop with 30 participants, three focus groups with 19 participants, and a validation workshop with 30 participants, this paper presents insights from practitioners at six large-scale companies, describing experiences from real industrial settings with regards to testing of software units and components. As every tool is also a cost for the organization, this paper presents a structured and holistic approach based on grounded theory, where methods and tools are evaluated and selected from four categories, representing test activities with different purposes: Optimize feedback loops, Improve test efficiency, Improve test suite effectiveness, and Technical debt management. The participants in the study also described how methods and tools used within their companies were often not well described in the company’s ways-of-working, which caused frustration and confusion. Feedback from the continuous integration and delivery pipeline must be understandable (describing what the problem is), actionable (describing what the developer needs to do), and explained (describing and motivating why this is important). The findings from this study provide actionable strategies for how testing of units and components can be conducted in a more structured way, which could help companies cut costs, save time and identify problems earlier.