An Exploratory Study on Automatic Architectural Change Analysis Using Natural Language Processing Techniques

Continuous architecture is vital for developing large, complex software systems and supporting continuous delivery, integration, and testing practices. Researchers and practitioners investigate models and rules for managing change to support architecture continuity. They employ manual techniques to analyze software change, categorizing the changes as perfective, corrective, adaptive, and preventive. However, a manual approach is impractical for analyzing systems involving thousands of artefacts as it is time-consuming, labor-intensive, and error-prone. In this paper, we investigate whether an automatic technique incorporating free-form natural language text (e.g., developers' communication and commit messages) is an effective solution for architectural change analysis. Our experiments with multiple projects showed encouraging results for detecting architectural messages using our proposed language model. Although architectural change categorization for the preventive class is moderate, the outcome for the random dataset is insignificant in general (around a 45% F1 score). We investigated the causes of the unpromising outcome. Overall, our study reveals that our automated architectural change analysis tool would be fruitful only if the developers provide considerable technical details in the commit messages or other text.