Leveraging LLM-based data augmentation for automatic classification of recurring tasks in software development projects

Abstract

Background:

Issue tracking systems (ITS) store project task data that is valuable for analytics and simulation. Projects typically include two types of tasks: stateful and recurring. While stateful tasks can be automatically categorized with relative ease, categorizing recurring tasks remains challenging. Prior research indicates that a key difficulty may lie in the underrepresentation of certain task types, which leads to severely imbalanced training datasets and hampers the accuracy of machine-learning models for task categorization.

Aims:

The goal of this study is to evaluate whether leveraging large language models (LLM) for data augmentation can enhance the machine-learning-based categorization of recurring tasks in software projects.

Method:

We conduct our study on a dataset from six industrial projects comprising 9,589 tasks. To address class imbalance, we up-sample minority classes during training via data augmentation using LLMs and several prompting strategies, assessing their impact on prediction quality. For each project, we perform time-series 5-fold cross-validation and evaluate the classifiers using state-of-the-art metrics — Accuracy, Precision, Recall, F1-score, and MCC — as well as practice-inspired metric called Monthly Classification Error (MCE) that assess the impact of task misclassification on project planning and resource allocation. Our machine-learning pipeline employs Transformer-based sentence embeddings and XGBoost classifiers.

Results:

The model automatically classifies software process tasks into 14 classes, achieving MCC values between 0.71 and 0.76. We observed higher prediction quality for the largest projects in the dataset and for those managed using “traditional” project management methodologies. Moreover, employing intra-project data augmentation strategies reduced the MCE error by up to 43%.

Conclusions:

Our findings indicate that large language models (LLMs) can be used to mitigate the impact of imbalanced task categories, thereby enhancing the performance of classification models even with limited training data.