Improved Flaky Test Detection with Black-Box Approach and Test Smells

Abstract

Flaky tests can pose a challenge for software development, as they produce inconsistent results even when there are no changes to the code or test. This leads to unreliable results and makes it difficult to diagnose and troubleshoot any issues. In this study, we aim to identify flaky test cases in software development using a black-box approach. Flaky test cases are unreliable indicators of code quality and can cause issues in software development. Our proposed model, Fast-Flaky, achieved the best results in the cross-validation results. In the per-project validation, the results showed an overall increase in accuracy but decreased in other metrics. However, there were some projects where the results improved with the proposed pre-processing techniques. These results provide practitioners in software development with a method for identifying flaky test cases and may inspire further research on the effectiveness of different pre-processing techniques or the use of additional test smells.