Existence of nontrivial solutions for a fractional \(p\&q\)-Laplacian equation with sandwich-type and sign-changing nonlinearities

In this paper, we deal with the following fractional $p\&q$ -Laplacian problem: $$ \left \{ \textstyle\begin{array}{l@{\quad }l} (-\Delta )_{p}^{s}u +(-\Delta )_{q}^{s}u =\lambda a(x)|u|^{\theta -2}u+ \mu b(x)|u|^{r-2}u&\text{in}\;\ \Omega , \\ u(x)=0 &\text{in}\;\ \mathbb{R}^{N}\setminus \Omega , \end{array}\displaystyle \right . $$ where $\Omega \subset \mathbb{R}^{N}$ is a bounded domain with smooth boundary, $s\in (0,1)$ , $(-\Delta )_{m}^{s}$ $(m\in \{p,q\})$ is the fractional m-Laplacian operator, $p,q,r,\theta \in (1,p_{s}^{*}]$ , $p_{s}^{*}=\frac{Np}{N-sp}$ , $\lambda , \mu >0$ , and the weights $a(x)$ and $b(x)$ are possibly sign changing. Using the concentration compactness principle for fractional Sobolev spaces and the Ekeland variational principle, we prove that the problem admits a nonnegative solution for the critical case $r=p_{s}^{*}$ . Moreover, for the subcritical case $r< p_{s}^{*}$ , we obtain two existence results by applying the Ekeland variational principle and the mountain pass theorem.