Existence of the fully nontrivial ground state solutions for the coupled nonlinear Brezis-Nirenberg Maxwell system

In this paper, we explore the existence of fully nontrivial solutions to the following nonlinear Brezis-Nirenberg Maxwell system$\{\begin{array}{cc}\nabla \times \nabla \times E_1+{\lambda }_1{E}_1={\kappa }_1|E_1{|}^{p-2}{E}_1+\beta |E_2{|}^2{E}_1,& \text{in}\Omega ,\\ \nabla \times \nabla \times E_2+{\lambda }_2{E}_2={\kappa }_2|E_2{|}^{p-2}{E}_2+\beta |E_1{|}^2{E}_2,& \text{in}\Omega ,\\ {\nu }_1\times E_1=0,{\nu }_2\times E_2=0,& \text{on}\partial \Omega ,\end{array}$ where ${\lambda }_i\le 0,{\kappa }_i>0(i=1,2),\beta >0$, $p\in (2,6]$, and $\Omega \subset R^3$ is a simply connected, smooth, bounded Lipschitz domain with connected boundary, and ${\nu }_1,{\nu }_2:\partial \Omega \to R^3$ are the exterior normal. This system originates from the time-harmonic Maxwell equations and possesses a variational structure. We address both general subcritical cases and Sobolev critical cases, establishing the existence of a fully nontrivial ground state solution with cylindrical symmetry. Additionally, we prove several properties of these solutions. To achieve this, we develop a new critical point theory that not only resolves the current problem but also facilitates the treatment of more general anisotropic media and other variational problems. Notably, our results provide a positive answer to the open problem posed by T. Bartsch and J. Mederski in [7, Page 982, Problem 3]. Moreover, from a purely mathematical perspective, we extend the partial results from $N=3$ to the case of $N=4$.