Anomalous open orbits in Hofstadter spectrum of Chern insulator

The nontrivial band topology of Chern insulator can enrich the content of its Hofstadter butterfly spectra, which is closely related to recent experiments in twisted bilayer graphene and cold atom systems. We investigate the Hofstadter spectrum for various models of Chern insulators under a rational flux \({\phi_{0}}\over{q}\), here \(\phi_{0}={h\over{e}}\) and q being an integer. We find the number of splitting subbands is ∣q − C∣ with C denoting the Chern number of parent band. Importantly, anomalous open-orbital subbands with Chern numbers q − 1 and −q − 1 emerge unexpectedly, which are beyond the parameter window (−q/2, q/2) of the Diophantine equation studied by Thouless-Kohmoto-Nightingale-den Nijs [Phys. Rev. Lett. 49, 405 (1982)]. The emergence of anomalous open orbits can be traced by analyzing the evolution of the Hofstadter spectrum, identifying as a new type of topological phase transition. These novel findings not only pinpoint the peculiar Hofstadter spectrum of Chern insulator, but also provide routes to study exotic characteristics beyond the Landau level physics.