How is exploratory testing used? A state-of-the-practice survey

Abstract

Context: Exploratory Testing has experienced a rise in popularity in the industry with the emergence of agile development practices, yet it remains unclear, in which domains and how it is used in practice. Goal: To study how software engineers understand and apply the principles of exploratory testing, as well as the specific advantages and difficulties they experience. Method: We conducted an online survey in the period June to August 2013 among Estonian and Finnish software developers and testers. Results: Our main findings are that the majority of testers, developers, and test managers using ET, (1) apply ET to usability-critical, performance-critical, security-critical and safety-critical software to a high degree; (2) use ET very flexibly in all types of test levels, activities, and phases; (3) perceive ET as an approach that supports creativity during testing and that is effective and efficient; and (4) find that ET is not easy to use and has little tool support. Conclusions: The high degree of application of ET in critical domains is particularly interesting and indicates a need for future research to obtain a better understanding of the effects of ET in these domains. In addition, our findings suggest that more support to ET users should be given (guidance and tools).