Ground state solution for a generalized Choquard Schr $$\ddot{\text {o}}$$ dinger equation with vanishing potential in homogeneous fractional Musielak Sobolev spaces

This paper aims to establish the existence of a weak solution for the following problem:

$$\begin{aligned} (-\Delta )^{s}_{\mathcal {H}}u(x) +V(x)h(x,x,|u|)u(x)=\left( \int _{{\mathbb R}^{N}}\dfrac{K(y)F(u(y))}{|x-y|^\lambda }\,\textrm{d}y\right) K(x)f(u(x)), \end{aligned}$$

in \({\mathbb R}^{N}\) where \(N\ge 1\), \(s\in (0,1), \lambda \in (0,N), \mathcal {H}(x,y,t)=\int _{0}^{|t|} h(x,y,r)r\ dr,\) \( h:{\mathbb R}^{N}\times {\mathbb R}^{N}\times [0,\infty )\rightarrow [0,\infty )\) is a generalized N-function and \((-\Delta )^{s}_{\mathcal {H}}\) is a generalized fractional Laplace operator. The functions \(V,K:{\mathbb R}^{N}\rightarrow (0,\infty )\), non-linear function \(f:{\mathbb R}\rightarrow {\mathbb R}\) are continuous and \( F(t)=\int _{0}^{t}f(r)dr.\) First, we introduce the homogeneous fractional Musielak–Sobolev space and investigate their properties. After that, we pose the given problem in that space. To establish our existence results, we use variational technique based on the mountain pass theorem. We also prove the existence of a ground state solution by the method of Nehari manifold.