Sex determination is a fundamentally important and highly diversified biological process, yet the mechanisms behind the origin of this diversity are mostly unknown. Here we suggest that the evolution of sex determination systems can be driven by a chromosomal inversion. We show that an XY system evolved recently in particular nine-spined stickleback (Pungitius pungitius) populations, which arose from ancient hybridization between two divergent lineages. Our phylogenetic and genetic mapping analyses indicate that the XY system is formed in a large inversion that is associated with hybrid sterility between the divergent lineages. We suggest that a new male-determining gene evolved in the inversion in response to selection against impaired male fertility in a hybridized population. Given that inversions are often associated with hybrid incompatibility in animals and plants, they might frequently contribute to the diversification of sex determination systems.