Dynamic analysis of a drug resistance evolution model with nonlinear immune response

Recent studies have utilized evolutionary mechanisms to impede the emergence of drug-resistant populations. In this paper, we develop a mathematical model that integrates hormonal treatment, immunotherapy, and the interactions among three cell types: drug-sensitive cancer cells, drug-resistant cancer cells and immune effector cells. Dynamical analysis is performed, examining the existence and stability of equilibria, thereby confirming the model’s interpretability. Model parameters are calibrated using available prostate cancer data and literature. Through bifurcation analysis for drug sensitivity under different immune effector cells recruitment responses, we find that resistant cancer cells grow rapidly under weak recruitment response, maintain at a low level under strong recruitment response, and both may occur under moderate recruitment response. To quantify the competitiveness of sensitive and resistant cells, we introduce the comprehensive measures $R_1$ and $R_2$, respectively, which determine the outcome of competition. Additionally, we introduce the quantitative indicators $CI{E}_1$ and $CI{E}_2$ as comprehensive measures of the immune effects on sensitive and resistant cancer cells, respectively. These two indicators determine whether the corresponding cancer cells can maintain at a low level. Our work shows that the immune system is an important factor affecting the evolution of drug resistance and provides insights into how to enhance immune response to control resistance.