Reinforcement learning for automated method development in liquid chromatography: insights in the reward scheme and experimental budget selection

Abstract

Chromatographic problem solving, commonly referred to as method development (MD), is hugely complex, given the many operational parameters that must be optimized and their large effect on the elution times of individual sample compounds. Recently, the use of reinforcement learning has been proposed to automate and expedite this process for liquid chromatography (LC). This study further explores deep reinforcement learning (RL) for LC method development. Given the large training budgets required, an in-silico approach was taken to train several Proximal Policy Optimization (PPO) agents. High-performing PPO agents were trained using sparse rewards (=rewarding only when all sample components were fully separated) and large experimental budgets. Strategies like frame stacking or long short-term memory networks were also investigated to improve the agents further. The trained agents were benchmarked against a Bayesian Optimization (BO) algorithm using a set of 1000 randomly-composed samples. Both algorithms were tasked with finding gradient programs that fully resolved all compounds in the samples, using a minimal number of experiments. When the number of parameters to tune was limited (single-segment gradient programs) PPO required on average, 1 to 2 fewer experiments, but did not outperform BO with respect to the number of solutions found, with PPO and BO solving 17% and 19% of the most challenging samples, respectively. However, PPO excelled at more complex tasks involving a higher number of parameters. As an example, when optimizing a five-segment gradient PPO solved 31% of samples, while BO solved 24% of samples.