Reinforcement learning for cooling rate control during quenching

Abstract

Purpose

The premise of this research is that the coupling of reinforcement learning algorithms and computational dynamics can be used to design efficient control strategies and to improve the cooling of hot components by quenching, a process that is classically carried out based on professional experience and trial-error methods. Feasibility and relevance are assessed on various 2-D numerical experiments involving boiling problems simulated by a phase change model. The purpose of this study is then to integrate reinforcement learning with boiling modeling involving phase change to optimize the cooling process during quenching.

Design/methodology/approach

The proposed approach couples two state-of-the-art in-house models: a single-step proximal policy optimization (PPO) deep reinforcement learning (DRL) algorithm (for data-driven selection of control parameters) and an in-house stabilized finite elements environment combining variational multi-scale (VMS) modeling of the governing equations, immerse volume method and multi-component anisotropic mesh adaptation (to compute the numerical reward used by the DRL agent to learn), that simulates boiling after a phase change model formulated after pseudo-compressible Navier–Stokes and heat equations.

Findings

Relevance of the proposed methodology is illustrated by controlling natural convection in a closed cavity with aspect ratio 4:1, for which DRL alleviates the flow-induced enhancement of heat transfer by approximately 20%. Regarding quenching applications, the DRL algorithm finds optimal insertion angles that adequately homogenize the temperature distribution in both simple and complex 2-D workpiece geometries, and improve over simpler trial-and-error strategies classically used in the quenching industry.

Originality/value

To the best of the authors’ knowledge, this constitutes the first attempt to achieve DRL-based control of complex heat and mass transfer processes involving boiling. The obtained results have important implications for the quenching cooling flows widely used to achieve the desired microstructure and material properties of steel, and for which differential cooling in various zones of the quenched component will yield irregular residual stresses that can affect the serviceability of critical machinery in sensitive industries.