Software engineering meets legal texts: LLMs for auto detection of contract smells

Abstract

Although there have been many major advances in Artificial Intelligence including its application to a wide variety of tasks, some specialized domains remain difficult to tackle. In this work, we examine parallels between software engineering and legal contract drafting and analysis. Porting well-known code smells principles to various legal contracts, we introduce ”contract smells,” text patterns that are indicative of potentially significant issues within contractual agreements. We leverage semi-auto labeling with GPT-4, prompting and expert spot checks, to create datasets for suitability testing of auto detection of these contract smells. Using transformer-based models, we explore the impact of legal domain knowledge, hyperparameters fine tuning and specific task information on detection success. We achieve high accuracy with further fine-tuning of BERT as well as LEGAL-BERT, while more consistent results were achieved using task-specific data. We further demonstrate that although multi-class detection can boost coverage of rare smells, single-class detection yields better accuracy. While this is an initial foray into the idea of contract smells, this work underscores the feasibility of applying advanced NLP techniques and LLMs to automate aspects of legal contract review, suggesting a scalable path toward standardized, machine-assisted legal drafting and analysis.