A Review of Artificial Neural Network Applications in Variants of Optimization Algorithms

Abstract

The artificial neural network (ANN) is typically one of the most famous artificial intelligent (AI) field which inspired based on biological human brain model. The model of real human brain known as neurons have been transformed into a mathematical formulation, works as an artificial neuron that connected one to another in very systematical manners. Connected neurons imply an optimization notion as a major practice for training the neurons. Two main optimization problems; constraint and unconstraint have both viewed as a decision problem techniques use to find the best vector of decision variables over all possible vectors in certain optimization problems. The maximization of objectives function can be considered as main factor to determine convergence of trained neurons. Most popular of optimization algorithm such as Gradient, Newton’s, Conjugate or Quasi-Newton have shown different results depend on efficiency, accuracy, convergence time and overall performance based on the problem to be solved. The feedforward or backpropagation neural network models mostly apply the optimization algorithm as mentioned. Therefore, the goal of this paper is to review and study the difference types of optimization techniques used in neural network applications. Besides, the purpose of this review is also to give an overview of how optimization algorithms and its modified models have been applied and implemented in neural network training rule. Besides, this paper is intended to study in what manner optimization can change the execution of performance result and training analysis.