An optimized reinforcement learning based MTD mutation strategy for securing edge IoT against DDoS attack

Distributed Denial of Service (DDoS) attacks are among the most destructive and challenging threats to mitigate for computer networks, particularly in edge IoT environments. Moving Target Defense (MTD) is a promising security mechanism that undermines the adversary’s gathered information by dynamically altering the attack surface. A selection of network nodes is chosen for mutation, and these changes hinder the adversary from achieving their objectives. However, identifying the optimal set of nodes for effectively and efficiently mitigating a DDoS attack remains a significant challenge. Existing MTD approaches have only considered a single factor—either the node’s vulnerability level or connectivity—and often lack generality and scalability for real-world IoT implementations. In this paper, we propose an enhanced MTD approach called CVbMA (Connection- and Vulnerability-based MTD Approach) that jointly considers both the vulnerability levels and connection weights of nodes to inform mutation strategies. To ensure practical applicability and adaptability, we develop a cost-aware Reinforcement Learning (RL) framework that incorporates explicit mutation costs into the reward function and utilizes neural ranking and model compression for scalability. Extensive evaluations are conducted using both Mininet-based simulations and a physical IoT testbed with real attack traces and heterogeneous devices. Comprehensive benchmarking and ablation studies against state-of-the-art MTD baselines demonstrate that the proposed framework significantly reduces the adversary’s success rate and incidents of server crashes, while maintaining low overhead and achieving high adaptivity. A detailed analysis of real-world deployments highlights the robustness of systems under operational constraints, including fluctuating latency, hardware diversity, and asynchronous events. Limitations and future enhancements, including topology-aware RL, adaptive mutation scheduling, and continuous model updates, are discussed. The results affirm the practical, scalable, and robust potential of cost-sensitive RL-based MTD for next-generation IoT security.