Differentially Private Reward Functions in Markov Decision Processes: Policy Synthesis and Tradeoffs

Abstract

Policy synthesis in Markov decision processes uses a known reward function to compute a policy that maximizes it. However, onlookers may infer reward functions by observing agents, which can reveal sensitive information. Therefore, in this paper we introduce and compare two methods for privatizing reward functions in policy synthesis for multi-agent Markov decision processes, which generalize Markov decision processes. Reward functions are privatized using differential privacy, a statistical framework for protecting sensitive data. The methods we develop perturb (i) each agent's individual reward function or (ii) the joint reward function shared by all agents. We prove that both of these methods are differentially private and show approach (i) provides better performance. We then develop guidelines for selecting reward functions to achieve high performance while remaining private, and we quantify the increase in computation required to compute policies with privatized rewards. Numerical simulations are performed on several examples, and these examples show that even relatively strong differential privacy ($\epsilon =1.3$) induces as little as a 5% decrease in performance and 0.016% increase in computation time.