Excessive and asymmetrical removal of heterozygous sites by maxSH biases downstream population genetic inference: Implications for hybridization between two primroses

Techniques of reduced-representation sequencing (RRS) have revolutionized ecological and evolutionary genomics studies. Precise establishment of orthologs is a critical challenge for RRS, especially when a reference genome is absent. The proportion of shared heterozygous sites across samples is an alternative criterion for filtering paralogs. In the prevailing pipeline for variant calling of RRS data – PYRAD/IPYRAD, maxSH is an often overlooked parameter with implications to detecting and filtering paralogs according to shared heterozygosity. Using empirical genotyping by sequencing data of two primroses (Primula alpicola Stapf and Primula florindae Ward) and their putative hybrids, and extra data sets of Californian golden cup oaks, we explore the impact of maxSH on filtering paralogs and further downstream analyses. Our study sheds light on the simultaneous validity and risk of filtering paralogs using maxSH, and its significant effects on downstream analyses of outlier detection, population assignment, and demographic modeling, emphasizing the importance of attention to detail during bioinformatic processes. The mutual confirmation between results of population assignment and demographic modeling in this study suggested maxSH = 0.10 has a potentially excessive and asymmetrical effect on the removal of truly shared heterozygous sites as paralogs. These results indicate that hybridization origin hypotheses of putative hybrids represented by results with maxSH = 0.25 and 0.50 are more credible. In conclusion, we revealed the critical hazard of paralogs filtration according to sharing heterozygosity at first, so that we propose to use specific protocols, rather than maxSH, to filter potential paralogs for closely related lineages.