A Comparative Analysis of Hold Out, Cross and Re-Substitution Validation in Hyper-Parameter Tuned Stochastic Short Term Load Forecasting

Abstract

Analysis of load plays an important role in the operation of modern power systems due to its highly intermittent nature. This manuscript proposes the best approach by comparing results of Hold out, Cross and Re-Substitution validation from hyperparameter tuned Short Term Load Forecasting (STLF). Tree, Neural Network and GPR (Gaussian Process Regression) are three stochastic regression methods used. Each validation procedure is compared with every considered regression method, leading to 9 such combinations. Each combination is analysed with statistical parameters like RMSE (Root Mean Square Error), R Squared, MSE(Mean Square Error), MAE (Mean Absolute Error) and training time. The best approach is further optimised by modifying hyper parameters using Bayesian, Grid Search, and Random Search and most suitable method is proposed. The simulations are performed in Python and MATLAB platforms. The best combination for computation of STLF is K-fold validation with Tree Regression The statistical parameters obtained from the combination are RMSE, R Squared, MSE, MAE and training time of 0.077, 0.88, 0.0059, 0.046, 1.2 respectively. The best method for hyper-parameter tuning is found out to be Grid search with a reduced MSE of 0.0023.