Memory Capacity of Neural Networks with Threshold and Rectified Linear Unit Activations

Abstract

Overwhelming theoretical and empirical evidence shows that mildly overparametrized neural networks---those with more connections than the size of the training data---are often able to memorize the ...