LISA: A Transferable Light-weight Multi-Head Self-Attention Neural Network Model for Lithium-Ion Batteries State-of-Charge Estimation

Deep learning has drawn wide attention in state-of-charge (SOC) estimation for batteries in electric vehicles, owning to its model-free feature and powerful ability of nonlinear regression. However, previous studies focused on improving estimation accuracy, with less research devoted to computational efficiency and hardware implementation. This paper proposes a light-weight multi-head self-attention neural network with only 298 trainable parameters and 1.18K floating point operations (FLOPs), while achieving 2.10% RMSE at all conditions. This proposed self-attention and simple Recurrent Neural Networks (RNN) can operate independently in the battery model, and can be greatly accelerated by on-chip parallel execution. Furthermore, a transfer learning (TL) method was applied to the proposed model. The proposed model with transfer learning performs significantly better on small training subsets and stopped with fewer epochs using an early stopping strategy. To the best of our knowledge, it is the first model using Transformer and self-attention for SOC estimation.