Empirically driven SE research: state of the art and required maturity

Software engineering researchers are increasingly relying on the empirical approach to advance the state of the art. The level of empirical rigor and evidence required to guide software engineering research, however, can vary drastically depending on many factors. In this session we identify some of these factors through a discussion of the state of the art in performing empirical studies in software engineering, and we show how we can utilize the notion of empirical maturity to set and adjust the empirical expectations for software engineering research efforts.Regarding the state of the art in performing empirical studies, we will offer perspectives on two classes of study: those concerned with humans utilizing a technology, e.g., a person applying a methodology, a technique, or a tool, where human skills and the ability to interact with the technology are some of the primes issues, and those concerned with the application of the technology to an artifact, e.g., a technique or tool applied to a design or a program. In the first case, the emphasis is typically on issues like feasibility, usefulness, and then on effectiveness. The technology tends to be less well specified and based more on the experience and skills of the technology applier. In the second case, the emphasis is typically on the efficiency and effectiveness of the technology. The technology tends to be well defined and the assumption is that the individual skill and experience plays a less important role. We will discuss the set of factors that influence the design, implementation, and validity of these studies.Regarding empirical maturity and its implications on the SE community's expectations, we will provide examples of the large spectrum of studies with different maturity levels that can be performed to successfully support software engineering research. We will then identify and analyze the following aspects that are likely to impact a study's maturity level: technology (well-specified vs. under development), goals of the study (effectiveness vs. feasibility), type of design and analysis (controlled experiment vs. case study, quantitative vs. qualitative), control and specification of threats to validity (internal vs. external threats), dependence on context (in vivo vs. in vitro), relationship to previous empirical work (replicated on-site, replicated off-site, non-replicated, non-replicable), and purposes of the study (exploratory vs. confirmatory). We will lead a discussion on these key aspects that must be considered to assess the empirical maturity of a piece of work in the context of its research area and the empirical maturity of that area.