XYOnion: a layer-based method for splitting datasets into calibration and validation subsets

Abstract

Background

Chemometric models are widely used in analytical chemistry to interpret complex multivariate data. However, they rely on robust validation, which typically involves splitting the dataset into calibration and validation subsets when an independent set is missing. Dataset splitting methods such as random splitting, Kennard-Stone, SPXY, and Onion each have limitations, including biased performance estimates or unbalanced subset distributions. This study introduces XYOnion, a novel algorithm that addresses these issues by combining the strengths of SPXY and Onion in a unified framework.

Results

XYOnion generates representative calibration and validation subsets by combining distances in both the predictor (X) and response (y) spaces, and assigning samples in layered shells based on this combined metric. This layered structure ensures balanced coverage across the entire data space and prevents extrapolation in the validation subset, leading to more robust model assessments. The algorithm was tested on both simulated and real datasets and systematically compared against commonly used methods such as random splitting, Kennard-Stone, SPXY, and Onion. Results demonstrate that XYOnion produces more realistic and stable figures of merit, effectively avoiding the overly optimistic performance estimates that arise from unbalanced or non-representative splits. Additionally, the incorporation of the DISTSLCT algorithm improves computational efficiency by eliminating the need to compute full pairwise distance matrices, thereby enhancing scalability and making XYOnion suitable for large and high-dimensional datasets encountered in chemometric applications.

Significance and novelty

In conclusion, XYOnion offers a practical and reliable approach for dataset splitting in situations where independent test sets are unavailable. By integrating distance information from both predictor and response variables with a layered sampling strategy, the method ensures balanced and representative calibration and validation subsets. This leads to more realistic, reproducible, and trustworthy model evaluations.