Performance analysis of activation functions in molecular property prediction using Message Passing Graph Neural Networks

Abstract

Deep learning has significantly advanced molecular property prediction, with Message-Passing Graph Neural Networks (MPGNN) standing out as an effective method. This study systematically evaluates the performance of ten activation functions — Sigmoid, Tanh, ReLU, Leaky ReLU, ELU, SELU, Softmax, Swish, Mish, and GeLU — using the MPGNN model on the QM9 dataset. It aims to identify the most suitable activation functions for training neural networks for specific molecular properties. The study examines electronic properties such as HOMO, LUMO, HOMO-LUMO energy gap, dipole moment, and polarizability, as well as thermal properties like zero-point vibrational energy and specific heat capacity. The findings reveal that different activation functions excel for different properties: SELU for HOMO, ELU for LUMO, Sigmoid for the HOMO-LUMO gap, Mish for polarizability, GELU for ZPVE, and Leaky ReLU for dipole moment and specific heat capacity. These insights are vital for optimizing MPGNN design for targeted molecular property prediction.