Crosses between populations or species often display an asymmetry in the fitness of reciprocal F1 hybrids. This pattern, referred to as isolation asymmetry or Darwin’s Corollary to Haldane’s rule, has been observed in taxa from plants to vertebrates, yet we still know little about which factors determine its magnitude and direction. Here we show that differences in offspring size predict the direction of isolation asymmetry observed in crosses between populations of a placental fish, Heterandria formosa. In crosses between populations with differences in offspring size, high rates of hybrid inviability occur only when the mother is from a population characterized by small offspring. Crosses between populations that display similarly sized offspring, whether large or small, do not result in high levels of hybrid inviability in either direction. We suggest this asymmetric pattern of reproductive isolation is due to a disruption of parent-offspring coadaptation that emerges from selection for differently sized offspring in different populations.