A new optimal multistep optimal homotopy asymptotic method to solve nonlinear system of two biological species

Abstract Recently solving integro-differential equations have been the focus of attention among many researchers in the field of mathematic and engineering. The aim of current study is to apply the well-known optimal homotopy asymptotic method (OHAM) on a specific and famous model of these equations. It is illustrated that auxiliary functions and the number of Taylor series terms affect the accuracy of the solution. Hence, at first a solution has been found with an acceptable error by OHAM. Then, it has been continued to attain a better solution by Multistep optimal homotopy asymptotic method. All these processes had improved the precision of the solution. Auxiliary polynomials of two, three, and four degrees and different numbers of Taylor series term have been investigated to solve a nonlinear system derived by two biological species ‎living together. Ultimately, appropriate results with auxiliary polynomials of degree four and Taylor series with six terms have been obtained precisely. In addition, the error values decrease significantly compared to the other cases.