Multi-objective optimization-based and fault localization-oriented test case generation for novice programs

Abstract

Online judgment (OJ) systems are capable of evaluating program results by automatically executing test cases, significantly improving the efficiency of traditional guidance approaches. Moreover, existing studies attempt to assist novices through automated fault localization techniques to provide feedback to novices, which can help them quickly find the location of faulty statements. Among them, spectrum-based fault localization (SBFL) techniques have been widely used for their lightweight and efficiency, which only requires coverage information and test results of test cases to conduct fault localization. However, manually constructing high-quality test cases for a large number of OJ questions is tough work to complete. To solve this problem, we propose the novice program-oriented Multi-Objective Optimization-Based Fault Localization-Oriented Test Case Generation (MFTCG) for automatically generating test inputs. Specifically, we use multi-objective optimization algorithms to evolve the test case in terms of both fault localization and faulty code detection capability. We conduct experiments with 8911 programs from the well-known public OJ platform AtCoder. The results show that our proposed approach MFTCG can achieve the best fault localization performance compared with existing automated test case generation approaches in most cases and can achieve the similar faulty code detection capability compared to manually designed test cases.