A policy-based Monte Carlo tree search method for container pre-marshalling

AbstractThe container pre-marshalling problem (CPMP) aims to minimise the number of reshuffling moves, ultimately achieving an optimised stacking arrangement in each bay based on the priority of containers during the non-loading phase. Given the sequential decision nature, we formulated the CPMP as a Markov decision process (MDP) model to account for the specific state and action of the reshuffling process. To address the challenge that the relocated container may trigger a chain effect on the subsequent reshuffling moves, this paper develops an improved policy-based Monte Carlo tree search (P-MCTS) to solve the CPMP, where eight composite reshuffling rules and modified upper confidence bounds are employed in the selection phases, and a well-designed heuristic algorithm is utilised in the simulation phases. Meanwhile, considering the effectiveness of reinforcement learning methods for solving the MDP model, an improved Q-learning is proposed as the compared method. Numerical results show that the P-MCTS outperforms all compared methods in scenarios where all containers have different priorities and scenarios where containers can share the same priority.KEYWORDS: Container pre-marshalling problemMonte Carlo tree searchMarkov decision processQ-learning algorithmAutomated container terminal AcknowledgementThis research was made possible with funding support from National Natural Science Foundation of China [72101203, 71871183], Shaanxi Provincial Key R&D Program, China [2022KW-02], and China Scholarship Council [grant number 202206290124].Disclosure statementNo potential conflict of interest was reported by the author(s).Data availability statementData sharing not applicable – no new data generated.Additional informationFundingThis work was supported by National Natural Science Foundation of China: [Grant Number 72101203, 71871183]; China Scholarship Council: [Grant Number 202206290124]; Shaanxi Provincial Key R&D Program, China: [Grant Number 2022KW-02].Notes on contributorsZiliang WangMr. Ziliang Wang, is a Doctoral student from School of Management in Northwestern Polytechnical University.Chenhao ZhouDr. Chenhao Zhou, is a Professor from School of Management in Northwestern Polytechnical University. Prior to this, he was a Research Assistant Professor in the Department of Industrial Systems Engineering and Management, National University of Singapore. His research interests are transportation systems and maritime logistics using simulation and optimization methods.Ada CheDr. Ada Che, is a Professor from School of Management in Northwestern Polytechnical University. He received the B.S. and Ph.D. degrees in Mechanical Engineering from Xi’an Jiaotong University in 1994 and 1999, respectively. Since 2005, he has been a Professor in School of Management in Northwestern Polytechnical University. His current research interests include transportation planning and optimisation, production scheduling, and operations research.Jingkun GaoMr. Jingkun Gao, is currently an Engineer with Northwest Electric Power Design Institute Co., Ltd. of China Power Engineering Consulting Group and he received the Master’s degree from School of Management in Northwestern Polytechnical University.