On the instability and stability of non-homogeneous fluid in a bounded domain under the influence of a general potential

We investigate the instability and stability of specific steady-state solutions of the two-dimensional non-homogeneous, incompressible, and viscous Navier–Stokes equations under the influence of a general potential $f$. This potential is commonly used to model fluid motions in celestial bodies. First, we demonstrate that the system admits only steady-state solutions of the form $\left(\rho ,V,p\right)=\left({\rho }_0,0,P_0\right)$, where $P_0$ and ${\rho }_0$ satisfy the hydrostatic balance condition $\nabla P_0=-{\rho }_0\nabla f$. Additionally, the relationship between ${\rho }_0$ and the potential function $f$ is constrained by the condition $\left({\partial }_y{\rho }_0,-{\partial }_x{\rho }_0\right)\cdot \left({\partial }_{x}f,{\partial }_{y}f\right)=0$, which allows us to express $\nabla {\rho }_0$ as $h\left(x,y\right)\nabla f$. Second, when there exists a point $\left(x_0,y_0\right)$ such that $h\left(x_0,y_0\right)>0$, we establish the linear instability of these solutions. Furthermore, we demonstrate their nonlinear instability in both the Lipschitz and Hadamard senses through detailed nonlinear energy estimates. This instability aligns with the well-known Rayleigh–Taylor instability. Our study significantly extends and generalizes the existing mathematical results, which have predominantly focused on the scenarios involving a uniform gravitational field characterized by $\nabla f=(0,g)$. Finally, we show that these steady states are linearly stable provided that $h\left(x,y\right)<0$ holds throughout the domain. Moreover, they exhibit nonlinear stability when $h\left(x,y\right)$ is a negative constant.