Analysis of second-order Markov reward models

Abstract

This paper considers the analysis of second-order Markov reward models. In these systems the reward accumulation during state sojourns is not deterministic, but follows a Brownian motion with a state dependent drift and variance parameter. We give the differential equations that describe the density function and the moments of the accumulated reward, and show the similarities compared to the first-order (ordinary) case. A randomization based numerical method is also presented which is numerically stable, has an error bound to control the precision, and allows the efficient analysis of large models. The computational cost of the proposed procedure is practically the same as the one of the analysis of first-order reward models, while the modeling power of second-order models is clearly larger.