Constrained negentropy optimisation (CoNE-opt): Using independent components to merge satellite data products

Abstract

Merging multiple uncertain datasets cancels random errors, resulting in a merged product that is equivalent to or better than the best individual parent product used. Merging proceeds via a weighted average of individual products, with weights derived using second-order error statistics (variance/covariance), which cannot fully capture skewed or heavy-tailed error structures, and often restrictive assumptions (e.g., zero error cross-correlation, ECC), which are frequently violated when products share retrieval algorithms or calibration data. We propose here a novel alternative for merging geophysical data, Constrained Negentropy Optimisation (CoNE-opt), which assumes the true dataset exhibits the greatest departure from Gaussianity, an assumption commonly invoked in performing Independent Component Analysis (ICA) to derive source signals. CoNE-opt maximises negentropy, which is a measure of non-Gaussianity, and uses it as the objective function while incorporating a linear error model constraint. This formulation allows joint estimation of the full error spectrum and the merging weights. The method is validated through synthetic experiments followed by merger of three global satellite-derived surface soil moisture products, SMAP, SMOS, and SMOS-IC. Comparison against reference datasets demonstrate that CoNE-opt outperforms existing merging alternatives that rely on second order statistics instead. Notably, CoNE-opt improves error magnitudes in the presence of high ECC and outliers, where most Ordinary Least Squares (OLS)-based merging methods struggle. Superior performance was observed in soil moisture merging, where the overall normalised RMSE equals 0.15 (vs. 0.34 for the best OLS-based alternative), consistently surpassing input products and existing methods. These findings support the potential of CoNE-opt as a robust framework for generating global soil moisture datasets, thereby enhancing land surface and hydrological modelling in data-sparse regions.