Flower color polymorphism is relatively uncommon in natural flowering plants, suggesting that maintenance of different color morphs within populations is difficult. To address the selective mechanisms shaping pollen-color dimorphism, pollinator preferences and reproductive performance were studied over three years in Epimedium pubescens in which some populations had plants with either green or yellow pollen (and anthers). Visitation rate and pollen removal and receipt by the bee pollinator (Andrena emeishanica) did not differ between the two color morphs. Compared to the green morph, siring success of the yellow morph’s pollen was lower, but that of mixtures of pollen from green and yellow morphs was lowest. This difference, corresponding to in and ex vivo experiments on pollen performance, indicated that pollen germination, rather than tube growth, of the green morph was higher than that of the yellow morph and was seriously constrained in both morphs if a pollen competitor was present. A rare green morph may invade a yellow-morph population, but the co-existence of pollen color variants is complicated by the reduced siring success of mixed pollinations. Potential pollen competition between morphs may have discouraged the maintenance of multiple phenotypes within populations, a cryptic mechanism of competitive exclusion.