When good fits fail: assessing the reliability of machine learning models for PSA CH4/CO2 process optimization

Abstract

This work investigates the application of machine learning (ML) models for predicting and optimizing the performance parameters purity and recovery of a CH₄/CO₂ separation process via Pressure Swing Adsorption (PSA). Several ML algorithms were trained and tested using datasets generated from a detailed phenomenological PSA model, and their optimization performance was benchmarked against a reference Pareto front obtained from the same detailed model. The study critically examines the reliability of common goodness-of-fit metrics from the training and testing phases as predictors of optimization accuracy. Results reveal that high fitting accuracy in the dataset does not guarantee accurate optimization outcomes, while models with comparatively poorer fitting during training may outperform more complex models in the optimization task. Furthermore, traditional global error metrics are shown to be insufficient predictors of optimization reliability, with segmented, range-based error analysis providing better insights. Despite these challenges, the Gradient Boosted Tree model delivered highly accurate Pareto fronts with a computational cost reduction of over 65% compared to the full phenomenological model. These findings underscore both the potential and the limitations of ML-assisted process optimization and highlight the need for more nuanced error evaluation strategies in surrogate-based optimization frameworks.