The policy gradient estimation of continuous-time hidden Markov decision processes

Abstract

Recently, gradient based methods have received much attention to optimize some dynamic systems with hidden information, such as routing problems of robotic systems. In this paper, we presented a process - continuous time hidden Markov decision process (CTHMDP), which can be used to model the robotic systems. For this process, the problem of policy gradient estimation is studied. Firstly, an approximation formula to the gradient is presented, then by using the uniformization method, we introduce an algorithm, which can be considered as an extension of gradient of partially observable Markov decision process (GPOMDP) algorithm to the continue time model. Finally, the convergence and error bound of the algorithm are considered.