Hybrid game-theoretic security assessment of cyber-physical power systems using partial-information multi-agent reinforcement learning

In this paper, we develop a Hybrid Non-zero-sum Multi-stage Partial-information Stochastic (HNMPS) game theoretic model for assessing the security of interdependent cyber-physical power systems (CPPS). In the cyber layer, HNMPS encapsulates the discrete dynamics of attacks, with tactics and techniques defined in the ICS MITRE ATT&CK framework. In the physical layer, it evaluates the consequence of Denial of Service (DoS) attacks on the transient stability of power networks through non-linear continuous dynamic analysis. Next, we propose an Imperfect-Information Multi-Agent Q-learning (IMQL) algorithm to solve the game when opposing players' actions and strategies are unknown. Unlike existing methods, IMQL doesn't require joint Nash strategy computation and therefore relaxes the strong assumption on existing global optima or saddle point. We further prove the convergence of the proposed algorithm. Based on the outcome of the HNMPS, we introduce the Cyber Security Index Level (CIL), a novel metric that quantifies the probability of physical layer intrusion following a breach in the cyber layer. To validate our model, we conduct simulations on the Western System Coordinating Council (WECC) 9-bus system coupled with a cyber network, employing an attack scenario inspired by the BlackEnergy v3 malware. Results indicate the successful convergence and robustness of the learning process under partial information settings compared to existing algorithms.