EMERALD-O: efficient multi-agent reinforcement learning framework for optimised deep learning hyperparameter tuning and selection

Traditional hyperparameter tuning methods, such as Bayesian Optimization and Grid Search, often prove computationally expensive and inefficient for complex deep learning architectures. This paper introduces the Multi-Agent Reinforcement Learning (MARL) framework EMERALD-O to optimize deep learning networks. The MARL-based approach utilizes two specialized agents, Agent1 focuses on data augmentation and Agent 2 on managing the learning rate and optimizer selection. The agents operate within an environment that simulates the model’s training dynamics and uses validation accuracy as the reward signal. Agent performance is enhanced through epsilon-greedy exploration and experience replay mechanisms. EMERALD-O performs favorably 88.59 % with improved classification accuracy and training efficiency. The framework exhibits adaptability to diverse dataset characteristics, underscoring scalability and robustness. The framework was validated on different models built for image classification problem on Efficientnet, VGG16 and VGG19. The results highlight the potential of reinforcement learning to fine-tune complex neural network architectures and suggest that MARL can serve as a powerful tool to improve the performance of deep learning models. EMERALD-O can contribute by advancing the frontier of deep neural optimization, demonstrating that reinforcement learning can fundamentally transform the model-tuning approach. This framework establishes a new paradigm for automated hyperparameter optimization and provides a systematic lens for analyzing the behavior of the deep learning model across various hyperparametric configurations. By quantifying model responsiveness to parameter variations, this approach enables deeper insights into architectural characteristics and performance dynamics, facilitating both the theoretical understanding and practical optimization of deep learning systems.