Empirical studies on software evolution: should we (try to) claim causation?

Abstract

In recent and past years, there have been hundreds of studies aimed at characterizing the evolution of a software system. Many of these studies analyze the behavior of a variable over a given period of observation. How does the size of a software system evolve? What about its complexity? Does the number of defects increase over time or does it remain stable? In some cases, studies also attempt to correlate variables, and, possibly, to build predictors upon them. This is to say, one could estimate the likelihood that a fault occurs in a class, based on some metrics the class exhibits, on the kinds of changes the class underwent. Similarly, change couplings can be inferred by observing how artifacts tend to co-change. Although in many cases we are able to obtain models ensuring good prediction performances, we are not able to claim any causal-effect relationship between our independent and dependent variables. We could easily correlate the presence of some design constructs with the change-proneness of a software component, however the same correlation could be found with the amount of good Belgian beer our developers drink. As a matter of fact, the component could undergo changes for other, external reasons. Recent software evolution studies rely on fine-grained information mined by integrating several kinds of repositories, such as versioning systems, bug tracking systems, or mailing lists. Nowadays, many other precious sources of information, ranging from code search repositories, vulnerability databases, informal communications, and legal documents are also being considered. This would possibly aid to capture the rationale of some events occurring in a software project, and link them to statistical relations we observed. The road towards shifting from solid empirical models towards "principles of software evolution" will likely be long and difficult, therefore we should prepare ourselves to traverse it and go as far as possible with limited damages. To do this, we need to carefully prepare our traveling equipment by paying attention at: (i) combining quantitative studies with qualitative studies, surveys, and informal interviews, (ii) relating social relations among developers with variables observed on the project, (iii) using proper statistical and machine learning techniques able to capture the temporal relation among different events, and (iv) making a massive use of natural language processing and text mining among the various sources of information available.