A $\mathcal{K}$-Divergence Based Approach for Robust Regression Analysis

Abstract

This paper deals with the problem of robust regression analysis in the presence of outliers in both input and output data sets. In this context, we infer the input-output relation of a system by minimizing a new robust loss between the outputs and a presumed parametric function of the inputs. The considered loss is derived from an empirical estimate of a non-trivially modified version of the recently developed $\mathcal{K}$-divergence (adapted here for regression analysis). This modified version utilizes a model-free data-weighting mechanism based on Parzen’s non-parametric $\mathcal{K}$ernel density estimator, associated with the underlying joint distribution of the input and output data. The considered Parzen’s estimator involves two strictly positive smoothing “$\mathcal{K}$”ernel functions. These are defined independently across the input and output domains with possibly different bandwidth parameters. This data-weighting strategy leads to mitigation of low-density contaminations attributed to different types of input and output outlying measurements. The considered approach is illustrated for robust training of GELU neural networks, with applications to function approximation and time-series prediction.