Are Static Analysis Violations Really Fixed? A Closer Look at Realistic Usage of SonarQube

The use of automatic static analysis tools (ASATs) has gained increasing attention in the last few years. Even though available research have already explored ASATs issues and how they are fixed, these studies rely on revisions of the software, instead of mining real usage of these tools and real issue reports. In this paper we contribute with a comprehensive, multi-method study about the usage of SonarQube (a popular static analysis tool), mining 421,976 issues from 246 projects in four different instance of SonarQube: two hosted in open-source communities (Eclipse and Apache) and two hosted in Brazilian government institutions (Brazilian Court of Account (TCU) and Brazilian Federal Police (PF)). We first surveyed team leaders of the analyzed projects and found that they mostly consider ASATs warning messages as relevant for overall software improvement. Second, we found that both Eclipse and TCU employ highly customized instance of SonarQube, with more than one thousand distinct checkers–though just a subset of these checkers actually led to issues' reports. Surprisingly, we found a low resolution rate per project in all organizations–on average, 13% of the issues have been solved in the systems. We conjecture that just a subset of the checkers reveal real design and coding flaws, and this might artificially increase the technical debt of the systems. Nevertheless, considering all systems, there is a central tendency(median) of fixing issues after 18.99 days they had been reported, faster than the period for fixing bugs as reported in previous studies.