CLG: Automated checklist generation for improved pull request quality

Abstract

The Pull-Based development model, widely embraced in open-source software (OSS), leverages closer global collaboration. However, the increasing number of contributors introduces challenges, particularly in PR quality. Mature repositories address this by using checklists in Pull Request Templates (PRTs). Despite their benefits, a mere 14.15% of popular GitHub repositories implement such checklists. To address this gap, we propose CLG, a Check-List Generation approach, utilizing techniques with a multi-label classifier and summary generation to automatically generate checklists from contributing guidelines. Evaluation results demonstrate CLG’s superiority in each sub-task. As for categorize paragraphs of contributing guidelines, CLG improves 9.81%, 8.44%, 11.84%, and 10.17% across metrics of Accuracy, Precision, Recall, and F1-score. In the task of note generation, CLG improves 16.33%, 15.80%, and 6.43% in terms of ROUGE metrics. To investigate our generated checklists from actual perspectives, we submit our results to the open-source community. The results show that our approach can help the majority of repository maintainers and the open-source community in managing and submitting their received PRs.