Polygenic barriers to sex chromosome turnover

In organisms with distinct sexes, sex is commonly determined by segregation of a pair of chromosomes in paternal meiosis (male heterogamety) or maternal meiosis (female heterogamety). Evolutionary transitions between sex-chromosome systems—which change the chromosome pair that determines sex, the system of heterogamety, or both—are frequent in some clades. In others, however, sex chromosome systems show long-term stability. Previous explanations of this stasis rely on evolutionary dynamics peculiar to sex linkage, such as the accumulation of deleterious mutations on the sex-specific chromosome or sexually antagonistic mutations on either sex chromosome. Here, I show that the ordinary operation of stabilizing selection on quantitative traits can, as a by-product, promote stability of sex chromosome systems. The reason is that stabilizing selection on a trait permits individual chromosomes' genetic contributions to the trait to drift upwards or downwards, subject only to the constraint that they all sum to the trait optimum. This chromosome-specific drift causes reduced fitness in individuals with unusual chromosomal constitutions, such as the novel sexual genotypes that sex chromosome turnovers always produce. The theory can explain a suite of patterns in the phylogenetic distribution of sex determining systems. Applied to interspecific hybrids, it can also explain two key observations concerning their reduced fitness: Haldane's rule and the large-X effect.