Automated Penetration Testing with Fine-Grained Control through Deep Reinforcement Learning

Abstract

Penetration testing (PT) is an active method of evaluating the security of a network by simulating various types of cyber attacks in order to identify and exploit vulnerabilities. Traditional PT involves a time-consuming and labor-intensive process that is prone to errors and cannot be easily formulated. Researchers have been investigating the potential of deep reinforcement learning (DRL) to develop automated PT (APT) tools. However, using DRL in APT is challenged by partial observability of the environment and the intractability problem of the huge action space. This paper introduces RLAPT, a novel DRL approach that directly overcomes these challenges and enables intelligent automation of the PT process with precise control. The proposed method exhibits superior efficiency, stability, and scalability in finding the optimal attacking policy on the simulated experiment scenario.