CtrlFuzz: A controllable diffusion-based fuzz testing for deep neural networks via coverage-aware manifold guidance

Context:

Deep neural networks (DNNs) have been extensively deployed in safety-critical applications. Nevertheless, the inherent vulnerability to subtle perturbations of inputs constitutes serious risks to the reliability of DNN-based systems. While mutation-based coverage-guided fuzzing (CGF) ensures test oracle through deliberately limited perturbations, it struggles to obtain diverse and sparse test cases. Conversely, generation-based CGF is able to create more diverse test cases aligned with data distribution but lacks precise controllability.

Objective:

To refine the controllability and effectiveness of CGF in DNN testing, we aim to design a framework that is capable of generating realistic test cases with fine-grained control, while systematically exploring model vulnerabilities through a manifold-aware coverage criterion.

Method:

In this paper, we propose CtrlFuzz, a manifold coverage-guided controllable diffusion framework for testing DNNs. CtrlFuzz leverages manifold learning to embed high-dimensional inputs into a lower-dimensional Euclidean space, preserving geometric structure. Based on this, we define a manifold coverage by quantifying the ratio between the distances from seed and the non-adversarial counterparts to class center. We further enhance the testing controllability via performing semantic decomposition on seed inputs. A customized diffusion model based on the U-Net structure integrates manifold coverage and semantic constraints into the denoising process, which allows to remain semantically natural while covering vulnerable regions.

Results:

Experimental results on four popular datasets and ten benchmark DNN architectures demonstrate that CtrlFuzz (1) effectively maintains the semantic coherence of generated test cases, (2) achieves improved exploration of vulnerable manifold regions compared to existing CGF techniques, and (3) discovers significantly more error-inducing inputs on multiple model types.

Conclusion:

CtrlFuzz introduces a novel manifold guiding and diffusion-based fuzzing for controllable test case synthesis. By enhancing both manifold coverage and controllability in CGF, CtrlFuzz improves the thoroughness and effectiveness of DNN testing, which offers a promising direction for future robustness evaluation frameworks.