{"title":"Zeroth-Order Actor–Critic: An Evolutionary Framework for Sequential Decision Problems","authors":"Yuheng Lei;Yao Lyu;Guojian Zhan;Tao Zhang;Jiangtao Li;Jianyu Chen;Shengbo Eben Li;Sifa Zheng","doi":"10.1109/TEVC.2025.3529503","DOIUrl":"10.1109/TEVC.2025.3529503","url":null,"abstract":"Evolutionary algorithms (EAs) have shown promise in solving sequential decision problems (SDPs) by simplifying them to static optimization problems and searching for the optimal policy parameters in a zeroth-order way. Despite their versatility, EAs often suffer from high sample complexity due to neglecting underlying temporal structures. In contrast, reinforcement learning (RL) methods typically formulate SDPs as Markov decision process (MDP). Although more sample efficient than EAs, RL methods are restricted to differentiable policies and prone to getting stuck in local optima. To address these issues, we propose a novel evolutionary framework zeroth-order actor-critic (ZOAC). We propose to use stepwise exploration in parameter space and theoretically derive the zeroth-order policy gradient. We further utilize the actor-critic architecture to effectively leverage the Markov property of SDPs and reduce the variance of gradient estimators. In each iteration, ZOAC collects trajectories with parameter space exploration, and alternates between first-order policy evaluation (PEV) and zeroth-order policy improvement (PIM). We evaluate the effectiveness of ZOAC on a challenging multilane driving task optimizing the parameters in a rule-based, nondifferentiable driving policy that consists of three submodules: 1) behavior selection; 2) path planning; and 3) trajectory tracking. We also compare it with gradient-based RL methods on three Gymnasium tasks, optimizing neural network policies with thousands of parameters. Experimental results demonstrate the strong capability of ZOAC in solving SDPs. ZOAC significantly outperforms EAs that treat the problem as static optimization and matches the performance of gradient-based RL methods even without first-order information, in terms of total average return across tasks.","PeriodicalId":13206,"journal":{"name":"IEEE Transactions on Evolutionary Computation","volume":"29 2","pages":"555-569"},"PeriodicalIF":11.7,"publicationDate":"2025-01-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142981488","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Multiform Genetic Programming Framework for Symbolic Regression Problems","authors":"Jinghui Zhong;Junlan Dong;Wei-Li Liu;Liang Feng;Jun Zhang","doi":"10.1109/TEVC.2025.3527875","DOIUrl":"10.1109/TEVC.2025.3527875","url":null,"abstract":"genetic programming (GP) is a widely recognized and powerful approach for symbolic regression (SR) problems. However, existing GP methods rely on a single form to solve the problem, which limits their search diversity and increases the likelihood of getting stuck in local optima, especially in complex scenarios. In this article, we propose a general multiform GP (MFGP) framework to improve the performance of GP on complicated SR problems. As far as we know, this articel is the first attempt to integrate the multiform optimization paradigm with GP to accelerate the search performance. The key idea of the proposed framework is to construct multiple forms to solve the same problem cooperatively at the same time. During the evolution process, knowledge gained from different forms is shared among the solvers to improve the search diversity and efficiency. A knowledge transfer mechanism is specifically designed to facilitate knowledge transfer among GP solvers with different modeling forms. In addition, an adaptive resource control mechanism is designed to reallocate computing resources according to the problem solving efficiency of different solvers to further improve search efficiency. To demonstrate the effectiveness of the proposed framework, a multiform gene expression programming algorithm is designed and tested on 20 problems, including physical datasets, synthetic datasets, and real-world datasets. The experimental results have demonstrated the effectiveness of the proposed framework.","PeriodicalId":13206,"journal":{"name":"IEEE Transactions on Evolutionary Computation","volume":"29 2","pages":"429-443"},"PeriodicalIF":11.7,"publicationDate":"2025-01-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142940442","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Tai-You Chen, Wei-Neng Chen, Jin-Kao Hao, Yang Wang, Jun Zhang
{"title":"Multi-Agent Evolution Strategy With Cooperative and Cumulative Step Adaptation for Black-Box Distributed Optimization","authors":"Tai-You Chen, Wei-Neng Chen, Jin-Kao Hao, Yang Wang, Jun Zhang","doi":"10.1109/tevc.2025.3525713","DOIUrl":"https://doi.org/10.1109/tevc.2025.3525713","url":null,"abstract":"","PeriodicalId":13206,"journal":{"name":"IEEE Transactions on Evolutionary Computation","volume":"14 1","pages":""},"PeriodicalIF":14.3,"publicationDate":"2025-01-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142934498","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Lino Rodriguez-Coayahuitl, Ansel Y. Rodríguez-González, Daniel Fajardo-Delgado, Maria Guadalupe Sánchez Cervantes
{"title":"Problem Decomposition Strategies and Credit Distribution Mechanisms in Modular Genetic Programming for Supervised Learning","authors":"Lino Rodriguez-Coayahuitl, Ansel Y. Rodríguez-González, Daniel Fajardo-Delgado, Maria Guadalupe Sánchez Cervantes","doi":"10.1109/tevc.2025.3526581","DOIUrl":"https://doi.org/10.1109/tevc.2025.3526581","url":null,"abstract":"","PeriodicalId":13206,"journal":{"name":"IEEE Transactions on Evolutionary Computation","volume":"7 1","pages":""},"PeriodicalIF":14.3,"publicationDate":"2025-01-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142934421","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}