Invariant Correlation of Representation With Label

Abstract

The Invariant Risk Minimization (IRM) approach aims to address the security challenge of out-of-distribution robustness (domain generalization) by training a feature representation that remains invariant across multiple environments. However, in noisy environments, noise can distort invariant features, leading to different environment-specific losses. Current IRM-related methods such as IRMv1 and VREx underperform in these settings because they enforce uniform losses across environments. While environmental noise causes environment-specific losses, it does not alter the fundamental correlation between invariant representations and labels. Based on this observation, we propose ICorr (Invariant Correlation), which leverages this correlation to extract invariant representations in noisy settings. Unlike existing approaches, ICorr accommodates different environment-specific inherent losses while maintaining a necessary condition for identifying IRM classifiers. We present a detailed case study demonstrating why previous methods may lose ground while ICorr can succeed. Through a theoretical lens, particularly from a causality perspective, we illustrate that the invariant correlation of representation with label is a necessary condition for the optimal invariant predictor in noisy environments, where as the optimization motivations for other methods may not be. Furthermore, we empirically demonstrate the effectiveness of ICorr by comparing it with other domain generalization methods on various noisy datasets.