Perturbation regulated kernel regressors for supervised machine learning

This paper develops a kernel perturbation-regulated (KPR) regressor based on the errors-in-variables models. KPR offers a strong smoothing capability critical to the robustness of regression or classification results. For Gaussian cases, the notion of orthogonal polynomials is instrumental to optimal estimation and its error analysis. More exactly, the regressor may be expressed as a linear combination of many simple Hermite Regressors, each focusing on one (and only one) orthogonal polynomial. For Gaussian or non-Gaussian cases, this paper formally establishes a “Two-Projection Theorem” allowing the estimation task to be divided into two projection stages: the first projection reveals the effect of model-induced error (caused by under-represented regressor models) while the second projection reveals the extra estimation error due to the (inevitable) input measuring error. The two-projection analysis leads to a closed-form error formula critical for order/error tradeoff. The simulation results not only confirm the theoretical prediction but also demonstrate superiority of KPR over the conventional ridge regression method in MSE reduction.