Foxtsage vs. Adam: Revolution or evolution in optimization?

Optimisation techniques are crucial in neural network training, influencing predictive performance, convergence efficiency, and computational feasibility. Traditional Optimisers such as Adam offer adaptive learning rates but struggle with convergence stability and hyperparameter sensitivity, whereas SGD provides stability but lacks adaptiveness. We propose Foxtsage, a novel hybrid optimisation algorithm that integrates the FOX-TSA (for global search and exploration) with SGD (for fine-tuned local exploitation) to address these limitations. The proposed Foxtsage Optimiser is benchmarked against the widely used Adam Optimiser across multiple standard datasets, including MNIST, IMDB, and CIFAR-10. Performance is evaluated based on training loss, accuracy, precision, recall, F1-score, and computational time. The study further explores computational complexity and the trade-off between optimisation performance and efficiency. Experimental findings demonstrate that Foxtsage achieves a 42.03% reduction in mean loss (Foxtsage: 9.508, Adam: 16.402) and a 42.19% decrease in loss standard deviation (Foxtsage: 20.86, Adam: 36.085), indicating greater consistency and robustness in optimisation. Additionally, modest improvements are observed in accuracy (0.78%), precision (0.91%), recall (1.02%), and F1-score (0.89%), showcasing better generalization capability. However, these gains come at a significant computational cost, with a 330.87% increase in time mean (Foxtsage: 39.541 sec, Adam: 9.177 sec), raising concerns about practical feasibility in time-sensitive applications. By effectively combining FOX-TSA’s global search power with SGD’s adaptive stability, Foxtsage provides a promising alternative for neural network training. While it enhances performance and robustness, its increased computational overhead presents a critical trade-off. Future work will focus on reducing computational complexity, improving scalability, and exploring its applicability in real-world deep-learning tasks.