Examination of the genetic architecture of hybrid breakdown can provide insight into the genetic mechanisms of commonly observed isolating phenomena such as Haldane’s rule. We used line-cross analysis to dissect the genetic architecture of divergence between two plant species that exhibit Haldane’s rule for male sterility and rarity, Silene latifolia and Silene diclinis. We made 15 types of crosses, including reciprocal F1, F2, backcrosses, and later-generation crosses, grew the seeds to flowering, and measured the number of viable ovules, proportion of viable pollen, and sex ratio. Typically, Haldane’s rule for male rarity in XY animal hybrids is explained by interactions involving recessive X-linked alleles that are deleterious when hemizygous (dominance theory), while sterility is explained by rapid evolution of spermatogenesis genes (faster-male evolution). In contrast, we found that the genetic mechanisms underlying Haldane’s rule between the two Silene species did not follow these conventions. Dominance theory was sufficient to explain male sterility, but male rarity likely involved faster-male evolution. We also found an effect of the neo-sex chromosomes of S. diclinis on the extreme rarity of some hybrid males. Our findings suggest that the genetic architecture of Haldane’s rule in dioecious plants may differ from those commonly found in animals.