A diffusive predator-prey system with hunting cooperation in predators and prey-taxis: I global existence and stability

In this paper, we present and investigate a generalized reaction-diffusion system of predator-prey dynamics that incorporates prey-taxis and a hunting cooperation effect in predators, subject to homogeneous Neumann boundary conditions. This system describes a predator-prey interaction, in which the prey exhibit group defense mechanisms against their predators, and the predators cooperate to hunt these defended prey. The mechanism of prey is implemented through the (repulsive) prey-taxis term, which affects the diffusion rate of the predators, while the hunting cooperation effect of the predators towards their prey is implemented through the functional response. Moreover, this system incorporates generalized functional forms for the prey's growth rate, the predators' functional response and mortality rate, and the prey-tactic sensitivity, allowing for adaptation to various scenarios. We first establish that solutions of the time- and space-dependent system with such ecological characteristics exist globally and are bounded by estimating an associated weighted integral. Secondly, we investigate the constant coexistence state of the generalized system by introducing a constructed function that incorporates the prey's growth rate, the predators' functional response and mortality rate. Finally, we find some conditions yielding the local stability of all feasible constant and nonnegative solutions of the system, thereby revealing the occurrence of bistability. Furthermore, we conduct an investigation into the global stability at both the constant coexistence and predator-free states by applying Lyapunov stability analysis. We also analyze the rate at which the solutions to the system converge to these steady-states by utilizing the boundedness of the solutions along with Gagliardo-Nirenberg inequality.