Sex determination (SD) is an essential and ancient developmental process, but the genetic systems that regulate this process are surprisingly variable. Why SD mechanisms vary so much is a longstanding question in evolutionary biology. SD genes are generally located on sex chromosomes which also carry genes that interact epistatically with autosomes to affect fitness. How this affects the evolutionary stability of SD mechanisms is still unknown. Here, we explore how epistatic interactions between a sexually antagonistic (SA) non-SD gene, located on either an ancestral or novel sex chromosome, and an autosomal gene affect the conditions under which an evolutionary transition to a new SD system occurs. We find that when the SD gene is linked to an ancestral sex chromosomal gene which engages in epistatic interactions, epistasis enhances the stability of the sex chromosomes so that they are retained under conditions where transitions would otherwise occur. This occurs both when weaker fitness effects are associated with the ancestral sex chromosome pair or stronger fitness effects associated with a newly-evolved SD gene. However, the probability that novel SD genes spread is unaffected if they arise near genes involved in epistasis. This discrepancy occurs because on autosomes, SA allele frequencies are typically lower than on sex chromosomes. In our model, increased frequencies of these alleles contribute to a higher frequency of epistasis which may therefore more readily occur on sex chromosomes. Because sex chromosome-autosome interactions are abundant and can take several forms, they may play a large role in maintaining sex chromosomes.

Data were generated using an evolutionary genetic model written in R (available through https://github.com/MartijnSchenkel/EpistasisSexDetermination); processing and analysis files, including secondary data and resulting figures, can be found there as well.