MORL-IC: Multi-objective reinforcement learning approaches for analog integrated circuits optimization

Abstract

Analog and RF integrated circuit (IC) design requires the simultaneous optimization of multiple, conflicting objectives under highly nonlinear and tightly coupled constraints. While prior studies — including our own — have demonstrated the feasibility of applying multi-objective reinforcement learning (MORL) to analog circuit optimization, the specific impact of workflow-level design choices on convergence behavior, simulation cost, and Pareto-front characteristics has remained insufficiently explored. This paper reformulates the Multi-Objective Deep Deterministic Policy Gradient (MODDPG) approach not as a single fixed algorithm, but as a family of optimization workflows that share an identical multi-objective actor–critic learning core while systematically differing in their initialization strategy and environment evaluation mechanism. Within this unified formulation, three configurations are investigated: (i) a baseline MODDPG workflow with random initialization and direct SPICE evaluation, (ii) MODDPG-2, which employs analytically derived extreme solutions to guide early exploration, and (iii) MODDPG-3, which introduces an ANN-based pseudo-designer to generate boundary solutions directly from performance specifications. In addition, a Fully-ANN execution mode is examined, where an ANN-based pseudo-simulator replaces SPICE during policy learning to accelerate environment interaction. By preserving the same reinforcement learning architecture across all variants, the proposed framework isolates the effects of structured initialization and surrogate-based environments on optimization outcomes. The workflows are evaluated on three analog circuits (active-loaded differential amplifier, folded-cascode amplifier, and voltage comparator) and one RF circuit (CMOS cross-coupled LC oscillator), as well as on standard analytical benchmarks. Comparative results against NSGA-II and MOEA/D show that no single method universally dominates; however, the proposed workflows consistently reduce the number of required SPICE simulations by approximately 30%–75% while maintaining competitive Pareto-front quality. This efficiency gain indicates that the reinforcement-learning agent progressively acquires design intuition comparable to that of an experienced human designer—learning to avoid unpromising regions of the design space and focusing evaluations on high-value candidates. The results therefore demonstrate that, beyond the choice of learning algorithm, workflow-level design decisions critically shape how effectively RL can emulate expert design behavior, offering practical guidance for balancing solution quality and computational cost in automated analog and RF circuit design flows.