From Open Source to Industry: A Replication Study of AAA Test Structure in Envestnet

Abstract

The Arrange-Act-Assert (AAA) pattern, a unified structure for organizing unit tests, is widely recognized for better comprehension and maintenance of tests. However, its adoption in commercial software development remains insufficiently understood. This study investigates how the AAA pattern is practiced in industrial settings by examining test cases from Envestnet Inc. (ENV), adapting automated AAA tagging techniques, and comparing AAA-related issues between commercial and open-source software (OSS) projects. We enhanced a machine learning-based AAA tagging model, previously applied to OSS, to suit ENV domain-specific codebases. We then analyzed AAA adherence and issue distribution across three ENV projects, followed by developer interviews to uncover root causes and assess refactoring strategies. The adapted tagging model achieved F1 scores of 0.75, 0.77, and 0.73 for Arrangement, Action, and Assertion, respectively. Compared with seven open-source projects (25.17%), ENV projects exhibited a significantly lower prevalence of AAA issues (15.68%, $\boldsymbol{p}\,\mathbf{=\,0.034}$, Cohen’s $\boldsymbol{d}\,\mathbf{=\,1.64}$). Interviews with developers further illuminated the underlying causes of these violations—such as production–test co-evolution and unfamiliarity with certain testing APIs—and highlighted nuanced refactoring strategies that carefully balance structural consistency, code clarity, and test performance. These nuances, however, did not emerge in our earlier investigations of open-source projects. This study demonstrates that incorporating domain-specific data to fine-tune pre-trained models like CodeBERT, alongside context-aware features, enables accurate AAA analysis. ENV projects have significantly fewer AAA issues than OSS, with distinct patterns such as the absence of Multiple Acts and the prevalence of Assert Precondition. Developer interviews reveal practical motivations behind AAA deviations and inform context-sensitive refactoring strategies. These findings provide actionable guidance for improving test structure and developing automated refactoring tools in practice.