Probabilistic Approximation of Stochastic Time Series Using Bayesian Recurrent Neural Network

Abstract

In this brief, we investigate the approximation theory (AT) of Bayesian recurrent neural network (BRNN) for stochastic time series forecasting (TSF) from a probabilistic standpoint. Due to the cumulative dependencies present in stochastic time series, which are incompatible with the recurrent structure of BRNN and further complicate the analysis of AT, we first perform marginalization and transform the time series into a probabilistically equivalent latent variable model (LVM). Subsequently, we analyze the AT by evaluating the approximation error between the output mean of BRNN and that of the LVM, which are derived through Taylor expansion-based uncertainty propagation and distribution parameterization, respectively. Finally, leveraging the Khinchin’s law of large numbers, we study the convergence in probability of the sampling-based training algorithm, i.e., Bayes by Backprop (BBB), and prove that increasing the number of Monte Carlo samples in BBB leads to a convergence probability approaching one. Numerical simulations are conducted to demonstrate the validity of our results.