RPSigmoid: A Randomized Parameterization for a Sigmoidal Activation Function

Activation functions are integral components in neural networks because they are the calculations that each neuron performs on their inputted data before outputting it to the next neuron to help the neural network match the ground truth of the data sooner, thus converging faster. However, popular activation functions are not parameterized and those that are, have too few parameters, therefore lacking the ability to fully train the shape of the activation function. This paper introduces RPSigmoid, an activation function based on the Sigmoid function, and with four additional parameters which represent the vertical stretch factor, horizontal stretch factor, angularity, and slope of the asymptotes (which might be horizontal or oblique) of the sigmoidal curve. These parameters are randomized within a range before training and their values are updated along with other neural network parameters during backpropagation. Affirmative results of RPSigmoid present neural network training with a low-resource approach to yield impressive training results.