Many classic models of speciation incorporate assortative mating based on mating groups, such as plants with different flowering times, and they investigate whether an ecological trait under disruptive natural selection becomes genetically associated with the selectively neutral mating trait. It is well known that this genetic association is potently destroyed by recombination. In this note, we point out a more fundamental difficulty: if a “knife-edge” symmetry assumption of previous models is violated, then the mating trait is no longer neutral and sexual selection eliminates the polymorphism in the mating locus. This result strengthens the growing consensus that magic traits are the more likely route to nonallopatric speciation. We expand the model assuming also ecological selection on the mating trait and investigate the conditions for natural selection to overcome sexual selection and maintain mating polymorphism; we find that the combination of natural and sexual selection can cause also bistability of allele frequencies.