TK-SMF: Top-K Semantic Mask Fusion With Dynamic Budget and Recovery Against Adversarial Patch Attacks Under Semantic Fragmentation in Vision Models

Abstract

Deep learning-based vision models have severe security vulnerabilities against physical adversarial patch attacks. Previous studies propose explainability-based defense mechanisms (e.g., PatchOut and SALIUITL). These methods aim to recover the system by masking the regions that affect model decisions the most. However, these approaches generally assume that the adversarial effect is located in a single, continuous region (single mask selection). In this study, we empirically demonstrate the “Semantic Fragmentation” and saliency bleeding problems caused by multiple scattered patches in explainability maps. To overcome this issue, we propose the “Top-K Semantic Mask Fusion with Dynamic Budget (TK-SMF)” system. The proposed framework initially extracts explainability maps using LayerGradCAM, subsequently constructs a pool of candidate semantic segments via the Fast Segment Anything Model (FastSAM), and ultimately fuses the Top-K highest-risk masks within a strictly fixed area budget. This budget is anchored to a predefined worst-case threat assumption, eliminating the need to know the attacker's actual patch count during inference. We then restore the selected masked areas using OpenCV Inpainting to achieve model recovery. We evaluated the proposed framework on 38 distinct and challenging physical-world scenarios, encompassing diverse tactical vehicle categories under single- and multipatch attacks. Experimental results prove that TK-SMF breaks the targeted attack illusion with a 100% success rate. The results show that dynamic and flexible defense architectures exceed the limits of the single-mask assumption. They significantly improve model accuracy and reliability against complex physical world threats.