An analysis on the effects of evolving the Monte Carlo tree search upper confidence for trees selection policy on unimodal, multimodal and deceptive landscapes

Monte Carlo Tree Search (MCTS) is a best-first sampling/planning method used to find optimal decisions. The effectiveness of MCTS depends on the construction of its statistical tree, with the selection policy playing a crucial role. A particularly effective selection policy in MCTS is the Upper Confidence Bounds for Trees (UCT). While MCTS/UCT generally performs well, there may be variants that outperform it, leading to efforts to evolve selection policies for use in MCTS. However, these efforts have often been limited in their ability to demonstrate when these evolved policies might be beneficial. They frequently rely on single, poorly understood problems or on new methods that are not fully comprehended. To address these limitations, we use three evolutionary-inspired methods: Evolutionary Algorithm (EA)-MCTS, Semantically-inspired EA (SIEA)-MCTS as well as Self-adaptive (SA)-MCTS, which evolve online selection policies to be used in place of UCT. We compare these three methods against five variants of the standard MCTS on ten test functions of varying complexity and nature, including unimodal, multimodal, and deceptive features. By using well-defined metrics, we demonstrate how the evolution of MCTS/UCT can yield benefits in multimodal and deceptive scenarios, while MCTS/UCT remains robust across all functions used in this work.