Intraspecific variation in flower color is often attributed to pollinator-mediated selection, yet this mechanism cannot explain flower color polymorphisms in self-pollinating species. Indirect selection mediated via biotic and abiotic stresses could maintain flower color variation in these systems. The selfing forb, Boechera stricta, typically displays white flowers, but some individuals produce purple flowers. We quantified environmental correlates of flower color in natural populations. To disentangle plasticity from genotypic variation, we performed a multiyear field experiment in five gardens. In controlled conditions, we evaluated herbivore preferences and the effects of drought stress and soil pH on flower color expression. In natural populations, purple-flowered individuals experienced lower foliar herbivory than did their white-flowered counterparts. This pattern also held in the common gardens. Additionally, low-elevation environments induced pigmented flowers (plasticity), and the likelihood of floral pigmentation decreased with source elevation of maternal families (genetic cline). Viability selection favored families with pigmented flowers. In the laboratory, herbivores exerted greater damage on tissue derived from white- vs purple-flowered individuals. Furthermore, drought induced pigmentation in white-flowered lineages, and white-flowered plants had a fecundity advantage in the well-watered control. Flower color variation in selfing species is probably maintained by herbivory, drought stress, and other abiotic factors that vary spatially.