Evaluating effect of different dominance genotype encodings on genome-wide association studies and genomic selection.

Objective: The quantification of dominance effects varies across different models, and the appropriate coding in genetic analyses remains debated. This study investigated several proposed dominance encoding methods, evaluating their performance in genetic analyses.

Methods: Three datasets, each representing the breeds Duroc, Landrace, and Yorkshire, were used in this study. We assessed heritability, genetic effects, and prediction accuracy in genomic selection (GS), as well as significant loci and statistical power in genome-wide association studies (GWAS).

Results: In GS, correlations among additive effects and among total genetic effects across models were high (0.9 to 1) under different dominance encodings for most traits, while only the (0, 1, 0) and (0, 2p, 4p-2) encodings maintained high correlations for all traits. The average prediction accuracy of the additive-dominance model with the (0, 1, 0) encoding increased by 2.79% and 1.69%, respectively, compared to the (0, 1, 1) and (0, 2p, 4p-2) encodings for all traits. In GWAS, the (0, 1, 0) encoding had higher statistical power compared to the (0, 1, 1) and (0, 2p, 4p-2) encodings, especially for rare variants. Additionally, different dominance encodings identified independent and distinct significant loci.

Conclusion: The (0, 1, 0) encoding method generally outperforms the others in genetic analyses, while alternative encodings provide complementary insights into dominance effects. These findings provide valuable guidance for selecting dominance encodings in genetic analyses.