On a Rayleigh-Faber-Krahn Inequality for the Regional Fractional Laplacian

We study a Rayleigh-Faber-Krahn inequality for regional fractional Laplacian operators. In particular, we show that there exists a compactly supported nonnegative Sobolev function u0 that attains the infimum (which will be a positive real number) of the set {∫∫ {u>0}×{u>0} |u(x)− u(y)| |x− y| dxdy : u ∈ H̊(R), ∫ R u = 1, |{u > 0}| ≤ 1 } . Unlike the corresponding problem for the usual fractional Laplacian, where the domain of the integration is R × R, symmetrization techniques may not apply here. Our approach is instead based on the direct method and new a priori diameter estimates. We also present several remaining open questions concerning the regularity and shape of the minimizers, and the form of the Euler-Lagrange equations.