Genealogical asymmetry under the IM model and a two-taxon test for gene flow.

Abstract

Methods for detecting gene flow between populations often rely on asymmetry in the average length of particular genealogical branches, with the ABBA-BABA test being a well known example. Currently, asymmetry-based methods cannot be applied to a pair of populations and such analyses are instead performed using model-based methods. Here we investigate genealogical asymmetry under a two-population Isolation with Migration model. We focus on genealogies where the first coalescence event is between lineages sampled from different populations, as the external branches of these genealogies have equal expected length as long as there is no post-divergence gene flow. We show that unidirectional gene flow breaks this symmetry and results in the recipient population having longer external branches. We derive expectations for the probability of this genealogical asymmetry and propose a simple statistic (Am) to detect it from genome sequence data. Am provides a two-taxon test for gene flow that only requires a single unphased diploid genome from each population, with no outgroup information. We use analytic expectations and simulations to explore how recombination, unequal effective population sizes, bidirectional gene flow and background selection influence Am and find that the statistic provides unambiguous evidence for gene flow under a continent-island history. We estimate Am for genome sequence data from Heliconius butterflies and Odocoileus deer, generating results consistent with previous model-based analyses. Our work highlights a signal of gene flow overlooked to date and provides a method that complements existing approaches for investigating the demographic history of recently diverged populations.