Computation of the steady-state probability of Markov chain evolving on a mixed state space

Abstract

Abstract The partitioning algorithm is an iterative procedure that computes explicitly the steady-state probability of a finite Markov chain 𝑋. In this paper, we propose to adapt this algorithm to the case where the state space E := C ∪ D E:=C\cup D is composed of a continuous part 𝐶 and a finite part 𝐷 such that C ∩ D = ∅ C\cap D=\emptyset . In this case, the steady-state probability 𝜋 of 𝑋 is a convex combination of two steady-state probabilities π C \pi_{C} and π D \pi_{D} of two Markov chains on 𝐶 and 𝐷 respectively. The obtained algorithm allows to compute explicitly π D \pi_{D} . If π C \pi_{C} cannot be computed explicitly, our algorithm approximates it by numerical resolution of successive integral equations. Some numerical examples are studied to show the usefulness and proper functioning of our proposal.