Neural Arithmetic Logic Units with Two Transition Matrix and Independent Gates

Abstract

Neural Networks have traditionally been used to handle numerical information based on their training. However, they often struggle with systematic generalization, particularly when the numerical range during testing differs from that used in training. To tackle this issue, we propose an enhanced version of an existing architecture known as Neural Arithmetic Logic Units (NALU), incorporating Independent Gates. We refer to this new architecture as Neural Arithmetic Logic Units with Independent Gates (NALUIG), which can represent numerical values through linear activations. It employs primitive arithmetic operators, managed by learned gates that operate independently of the input, to differentiate weight matrices for both the adder and multiplier. Additionally, we introduce two new architectures: Neural Arithmetic Logic Unit with two Transition Matrices (NALU2M) and Neural Arithmetic Logic Unit with two Transition Matrices and Independent Gates (NALU2MIG). Our experiments demonstrate that the enhanced neural networks can effectively learn to perform arithmetic and numeric image classification from the Modified National Institute of Standards and Technology database (MNIST), achieving significantly lower error rates compared to other existing neural networks. This approach utilizes independent gates to represent numerical values as distinct neurons without introducing non-linearity. In this paper, we present improved results regarding numerical range generalization compared to the current state-of-the-art.