Accelerating Software Engineering Research Adoption with Analysis Bots

Abstract

An important part of software engineering (SE) research is to develop new analysis techniques and to integrate these techniques into software development practice. However, since access to developers is non-trivial and research tool adoption is slow, new analyses are typically evaluated as follows: a prototype tool that embeds the analysis is implemented, a set of projects is identified, their revisions are selected, and the tool is run in a controlled environment, rarely involving the developers of the software. As a result, research artifacts are brittle and it is unclear if an analysis tool would actually be adopted. In this paper, we envision harnessing the rich interfaces provided by popular social coding platforms for automated deployment and evaluation of SE research analysis. We propose that SE analyses can be deployed as analysis bots. We focus on two specific benefits of such an approach: (1) analysis bots can help evaluate analysis techniques in a less controlled, and more realistic context, and (2) analysis bots provide an interface for developers to "subscribe" to new research techniques without needing to trust the implementation, the developer of the new tool, or to install the analysis tool locally. We outline basic requirements for an analysis bots platform, and present research challenges that would need to be resolved for bots to flourish.