A reinforcement learning based fuzzing technique for binary programs vulnerabilities detection

Abstract

Binary programs are susceptible to vulnerabilities that can lead to unauthorized access, data breaches, and system damage. Fuzzing is a promising technique for identifying vulnerabilities in binary programs. However, fuzzing is time-consuming and inefficient as many seeds are random and recurrently executed. This paper proposes a novel approach called Vulnerable State Guided Fuzzing (VSGFuzz) employs a heuristic mechanism for generating seeds to optimize vulnerability detection. This mechanism assesses the vulnerable probability of each function within the target binary program and employs reinforcement learning to mutate seeds based on a comprehensive reward calculation algorithm considering vulnerable probability and coverage assessment. Experiments evaluating VSGFuzz compared with other typical methods on several datasets demonstrated the superiority of the proposed method over other methods.