Gamete dimorphism (anisogamy) defines the sexes in most multicellular organisms. Theoretical explanations for its maintenance usually emphasize the size-related selection pressures of sperm competition and zygote survival, assuming that fertilization of all eggs precludes selection for phenotypes that enhance fertility. In external fertilizers, however, fertilization is often incomplete due to sperm limitation, and the risk of polyspermy weakens the advantage of high sperm numbers that is predicted to limit sperm size, allowing alternative selection pressures to target free-swimming sperm. We asked whether egg size and ejaculate size mediate selection on the free-swimming sperm of Galeolaria caespitosa, a marine tubeworm with external fertilization, by comparing relationships between sperm morphology and male fertility across manipulations of egg size and sperm density. Our results suggest that selection pressures exerted by these factors may aid the maintenance of anisogamy in external fertilizers by limiting the adaptive value of larger sperm in the absence of competition. In doing so, our study offers a more complete explanation for the stability of anisogamy across the range of sperm environments typical of this mating system and identifies new potential for the sexes to coevolve via mutual selection pressures exerted by gametes at fertilization.