The evolution of sexual dimorphism will depend on how sexual, fecundity and viability selection act within each sex, with the different forms of selection potentially operating in opposing directions. We examined selection in the dioecious plant Silene latifolia using planted arrays of selection lines that differed in flower size (small vs. large). In this species a flower size/number tradeoff exists within each sex, and males produce smaller and more numerous flowers than females. Moreover, floral traits are genetically correlated with leaf physiology. Sexual selection favoring males in the small-flower line occurred via greater overlap in the timing of flower output between males from this line and females. Fecundity selection favored males with high flower production, as siring success was proportionate to pollen production. Viability selection opposed sexual selection, favoring males from the large-flower line. In females, fecundity and viability selection operated in the same direction, favoring those from the large-flower line via greater seed production and survival. These results concur with the pattern of floral sexual dimorphism. Together with previous results they suggest that the outcome of the different forms of selection will be environmentally dependent, and therefore help to explain variation among populations in sexually dimorphic traits.