Non-stationary value iteration for adaptive average control of piecewise deterministic Markov processes

Abstract

The main goal of this paper is to present a non-stationary value iteration scheme for the adaptive average control of Piecewise Deterministic Markov Processes (PDMPs), introduced by M.H.A. Davis in Davis (1984, 1993) as a family of continuous-time Markov processes punctuated by random jumps and with inter-jump movement driven by a deterministic flow. It is assumed in this paper that there are no boundary jumps. We study the adaptive average optimal control problem of PDMPs, considering that the jump intensity $\lambda$, the post-jump transition kernel $Q$, as well as the cost $C$ depend on an unknown parameter ${\beta }^{\ast }$. For a sequence of strongly consistent estimators $\left\{{\beta }_n^{\ast }\right\}$ of ${\beta }^{\ast }$ (that is, ${\beta }_n^{\ast }$ converge to ${\beta }^{\ast }$ almost surely) a non-stationary value iteration (depending on the current estimate ${\beta }_n^{\ast }$) is shown to be optimal for the long-run average control problem. We assume a total variation norm condition on the parameters $\lambda$ and $Q$ of the process (which generalizes the minorization condition considered in Costa, Dufour and Genadot (2024), resulting in a span-contraction operator. The paper concludes with a numerical example.