Fulde–Ferrell–Larkin–Ovchinnikov State in Perpendicular Magnetic Fields in Strongly Pauli-Limited Quasi-Two-Dimensional Superconductors

We examine the Fermi-surface effect called the nesting effect for the Fulde-Ferrell-Larkin-Ovchinnikov (FFLO) state in strongly Pauli-limited quasi-two-dimensional superconductors,focusing on the effect of three-dimensional factors, such as interlayer electron transfer, interlayer pairing, and off-plane magnetic fields including those perpendicular to the most conductive layers (hereinafter called the perpendicular fields). It is known that the nesting effect for the FFLO state can be strong in quasi-low-dimensional systems in which the orbital pair-breaking effect is suppressed by applying the magnetic field parallel to the layers. Hence, it has sometimes been suggested that it may not work for perpendicular fields. We illustrate that, contrary to this view, the nesting effect can strongly stabilize the FFLO state for perpendicular fields as well as for parallel fields when t_z is small so that the Fermi surfaces are open in the k_z-direction, where t_z denotes the interlayer transfer energy. In particular, the nesting effect in perpendicular fields can be strong in interlayer states. For example, in systems with cylindrical Fermi surfaces warped by t_z /= 0, interlayer states with Delta_{k} propto sin k_z exhibit mu_e H_c approx 1.65 Delta_{alpha 0} for perpendicular fields, which is much larger than typical values for parallel fields, such as mu_e H_c = Delta_s0 of the s-wave state and mu_e H_c approx 1.28 Delta_d0 of the d-wave state in cylindrical systems with t_z = 0. Here, mu_e and H_c are the electron magnetic moment and upper critical field of the FFLO state at T = 0, respectively, and Delta_{alpha 0} = 2 omega_c e^{- 1/lambda_alpha. We discuss the possible relevance of the nesting effect to the high-field superconducting phases in perpendicular fields observed in the compounds CeCoIn_5 and FeSe, which are candidates for the FFLO state.