Fractional concepts in neural networks: Enhancing activation functions

Abstract

This study explores the integration of fractional calculus in neural networks by introducing fractional order derivatives (FOD) as tunable parameters in activation functions, enabling diverse function adaptation. We evaluate these fractional activation functions across datasets and architectures, comparing them with traditional and novel functions to assess their effects on accuracy, computational efficiency, and memory usage. Findings indicate that fractional functions, especially fractional Sigmoid, can yield better performance, though challenges persist.