A penalized estimation of the variogram and effective sample size

Abstract

The variogram function plays a key role in modeling intrinsically stationary random fields, especially in spatial prediction using kriging equations. However, determining whether a computed variogram accurately fits the underlying dependence structure can be challenging. Current nonparametric estimators often fail to guarantee a conditionally negative definite function. In this paper, we propose a new valid variogram estimator, constructed as a linear combination of functions from a predefined class, ensuring it meets essential mathematical properties. A penalty coefficient is introduced to prevent overfitting, reducing spurious fluctuations in the estimated variogram. We also extend the concept of effective sample size (ESS), an important metric in spatial regression, to a nonparametric framework. Our ESS estimator is based on the reciprocal of the average correlation and is calculated using a plug-in approach, with the consistency of the estimator being demonstrated. The performance of these estimates is investigated through Monte Carlo simulations across various scenarios. Finally, we apply the methodology to rasterized forest images, illustrating both the strengths and limitations of the proposed approach.