A fully adaptive framework for continuous-state stochastic dynamic programming

Abstract

Approximate dynamic programming (ADP) carries out approximation of the future value function (FVF) to enable numerical solutions to dynamic programming (DP). Recent ADP methodologies often employ the design and analysis of computer experiment (DACE) techniques for the FVF approximation. Use of DACE-based ADP approach, however, creates a “chicken and egg” situation where we cannot collect the data for statistical modeling until we know the state space region, but we do not know the state space region until we collect the data. To overcome this dilemma, this paper introduces a sequential state space exploration (SSSE) approach to adaptively identify the state space region for the experimental design while also sampling useful data for the statistical model. In the proposed methodology, the SSSE approach works in tandem with an adaptive value function approximation (AVFA) algorithm that gradually grows the complexity of the statistical model as more data are observed. This novel SSSE-AVFA approach features a “fully adaptive dynamic programming” algorithm, which can automatically and appropriately identify the three critical components (state space region, sample size of the data, and statistical model structure) for FVF approximation, thereby eliminating the need for time-consuming trial-and-error computational runs that were previously required. The SSSE-AVFA approach is examined with a nine-dimensional inventory forecasting problem and is compared with fixed structure runs in which the state space region, sample size of the data, and statistical model structure are assumed in advance. Our proposed methodology ensured either that the established solutions could be more reasonable or that the modeling process could effectively save the computational effort. With its full adaptiveness in determining those critical components, the SSSE-AVFA approach has the potential to be more effective and efficient than the traditional methods in handling a wide range of real-world continuous-state DP problems.