Distilling Large Information Sets to Forecast Commodity Returns: Automatic Variable Selection or Hidden Markov Models?

We investigate the out-of-sample, recursive predictive accuracy for (fully hedged) commodity future returns of two sets of forecasting models, i.e., hidden Markov chain models in which the coefficients of predictive regressions follow a regime switching process and stepwise variable selection algorithms in which the coefficients of predictors not selected are set to zero. We perform the analysis under four alternative loss functions, i.e., squared and the absolute value, and the realized, portfolio Sharpe ratio and MV utility when the portfolio is built upon optimal weights computed solving a standard MV portfolio problem. We find that neither HMM or stepwise regressions manage to systematically (or even just frequently) outperform a plain vanilla AR benchmark according to RMSFE or MAFE statistical loss functions. However, in particular stepwise variable selection methods create economic value in out-of-sample meanvariance portfolio tests. Because we impose transaction costs not only ex post but also ex ante, so that an investor uses the forecasts of a model only when they increase expected utility, the economic value improvement is maximum when transaction costs are taken into account.