Ground state magnetic structure and magnetic field effects in the layered honeycomb antiferromagnet YbOCl

YbOCl is a representative member of the van der Waals layered honeycomb rare-earth chalcohalide $R\text{Ch}X$ ($R$ = rare earth; Ch = O, S, Se, and Te; and $X$ = F, Cl, Br, and I) family reported recently. Its spin ground state remains to be explored experimentally. We grew high-quality single crystals of YbOCl and conducted comprehensive thermodynamic, elastic, and inelastic neutron scattering experiments down to 50 mK. The experiments reveal an antiferromagnetic phase below 1.3 K which is identified as a spin ground state with an intralayer ferromagnetic and interlayer antiferromagnetic ordering. By applying sophisticated numerical techniques to a honeycomb (nearest-neighbor)–triangle (next-nearest-neighbor) model Hamiltonian which accurately describes the highly anisotropic spin system, we are able to simulate the experiments well and determine the diagonal and off-diagonal spin-exchange interactions. The simulations give an antiferromagnetic Kitaev term comparable to the Heisenberg one. The experiments under magnetic fields allow us to establish a magnetic field–temperature phase diagram around the spin ground state. Most interestingly, a relatively small magnetic field ($\sim 0.3$ to 3 T) can significantly suppress the antiferromagnetic order, suggesting an intriguing interplay of the Kitaev interaction and magnetic fields in the spin system. The present study provides fundamental insights into the highly anisotropic spin systems and opens a window to look into Kitaev spin physics in a rare-earth-based system.