Dynamic Watermarking for Finite Markov Decision Processes

Dynamic watermarking is an active intrusion detection technique that can potentially detect replay attacks, spoofing attacks, and deception attacks in the feedback channel for control systems. In this paper, we develop a novel dynamic watermarking algorithm for finite-state finite-action Markov decision processes. We derive a lower bound on the mean time between false alarms and an upper bound on the mean delay between the time an attack occurs and when it is detected. We further compute the sensitivity of the performance of the control system as a function of the watermark. We demonstrate the effectiveness of the proposed dynamic watermarking algorithm by detecting a spoofing attack in a sensor network system.