Asymptotic Properties of ReLU FFN Sieve Estimators

Abstract

Recently, machine learning algorithms have increasing become popular tools for economic and financial forecasting. While there are several machine learning algorithms for doing so, a powerful and efficient algorithm for forecasting purposes is the multi-layer, multi-node neural network with rectified linear unit (ReLU) activation function – deep neural network (DNN). Studies have demonstrated the empirical applications of DNN but have devoted less research to investigate its statistical properties which is mainly due to its severe nonlinearity and heavy parametrization. By borrowing tools from a non-parametric regression framework, sieve estimator, we first show that there exists such a sieve estimator for a DNN. We next establish three asymptotic properties of the ReLU network: consistency, sieve-based convergence rate, and asymptotic normality, and then validate our theoretical results using Monte Carlo analysis.