Towards one nonconvex connected Markov chain control problem. An approach to numerical solution

Abstract

The control problems for Markov chains in continuous time usually can be reduced to the solution of a system of ordinary differential equations, which is an analog of the dynamic programming equation. This system contains, as an obligatory element, the solution of the minimization problem on the set of admissible controls at each instant of time and the success in obtaining a numerical solution depends on how effective this optimization procedure is, because it must be performed for all MC states at any time. Actually, afterward the problem of integrating a system of differential equations is not difficult, since it does not require the extra high accuracy. In one of our previous works, we tried to obtain a solution to the problem of managing the dams system, however even with a small number of dams an increase in the number of states leads to a serious increase in the time of the program performance with usage of standard computational means. In this paper, we propose to divide the minimization procedures on the right-hand side for which an analytical solution has been obtained and the integration procedures, which made it possible to radically increase the number of states and thereby, to raise the accuracy of the solution of the problem as a whole.