Evaluation Studies of Software Testing Research in the Brazilian Symposium on Software Engineering

Experimentation is the traditional way of identifying cause-effect relationships in scientific research. Lately, there has been an increasing understanding that experiments and other forms of evaluation should be more thoroughly disseminated among computer science and, in particular, Software Engineering (SE) researchers. Software testing (ST) is an important SE topic, where experiments are particularly valuable: since cost constraints and high effectiveness goals are common within this subfield, cost/benefit characteristics have to be adequately evaluated to deem a specific approach useful or not. This paper reports on a historical perspective of the evaluation studies present in ST research published in the Brazilian Symposium on Software Engineering (SBES). The survey characterizes the software testing-related papers published in the 24-year history of SBES, investigates the types of evaluation presented in these studies - when they were presented at all - and whether the number of evaluations has increased over the years. Additionally, the survey also brings a preliminary characterization of the Brazilian software testing community that adopts SBES as a vehicle to publish its research. Results show that the number of papers that present evaluation studies have significantly increased over the years. However, on the downside, amongst the papers that described some kind of evaluation, only 20% performed more rigorous evaluations (e.g. experiments or case studies in the industrial context), whereas 80% described exploratory, less rigorous case studies.