Feature selection via risk-bound utility maximization

Abstract

The ultimate goal of supervised feature selection is to identify a feature subset that minimizes classification risk. Contemporary methods, however, often rely on heuristic or model-dependent proxy criteria that lack a direct theoretical connection to this fundamental objective. To bridge this gap, we introduce a new feature selection framework that directly optimizes a model-agnostic utility function grounded in statistical learning theory. Our approach defines the utility of a feature subset based on the 1-Wasserstein distance between class-conditional distributions. This metric is theoretically powerful as it can be used to construct an upper bound on the Bayes classification error, allowing us to construct a utility function that is a direct surrogate for this risk bound. We instantiate this framework with a subset search strategy that effectively captures feature interactions by maximizing this risk-bound utility. Extensive experiments on real-world datasets demonstrate that our method not only achieves state-of-the-art classification performance but also demonstrates superior robustness and interpretability, providing a principled and powerful alternative to traditional feature selection methods, confirming our framework’s theoretical soundness.