The conditional barycenter problem, its data-driven formulation and its solution through normalizing flows

A family of normalizing flows is introduced for selectively removing from a data set the variability attributable to a specific set of cofactors, while preserving the dependence on others. This is achieved by extending the barycenter problem of optimal transport theory to the newly introduced conditional barycenter problem. Rather than summarizing the data with a single probability distribution, as in the classical barycenter problem, the conditional barycenter is represented by a family of distributions labeled by the cofactors kept. The use of the conditional barycenter and its differences with the classical barycenter are illustrated on synthetic and real data addressing treatment effect estimation, super-resolution, anomaly detection and lightness transfer in image analysis.