1. Seed size is a functional trait with important fitness consequences that potentially extend throughout the life cycle of plants. Dithyrea californica experiences selection for larger seeds in post‐germination stages but it is still uncertain how environmental factors mediate the strength and the direction of natural selection on seed size. 2. D. californica represents a unique opportunity to investigate selection on seed size in natural conditions due to a persistent seed ring that stays attached to the root throughout the plant's life. This makes it possible to unearth plants at any stage and measure the size of the seed from which they originated. 3. We conducted a factorial experiment manipulating water availability and intraspecific competition using plants that naturally germinated in the wild. 4. Selection on seed size via survivorship was nil because all individuals survived to reproduce. The strength and the direction of selection on seed size via fecundity depended on water availability and conspecific density. 5. Contrary to our predictions, increasing conspecific density relaxed directional selection favoring larger seeds, but only in the wettest conditions and an increase in water availability strengthened it, but only at low density. A possible explanation of these counter‐intuitive results relies on the observed absence of survival selection and increased plant growth rates under high water and low density. 6. Larger seeds require more resources to construct and when this cost is taken into account; there is no overall fitness increase with seed size. This nicely follows the life history theory predictions for optimal seed size. At the evolutionary equilibrium, if seeds could be larger, per seed fitness would still increase, which is what we observed. But cost corrected fitness should be flat. Maternal fitness equals per seed fitness times seed number, so any increase to per seed fitness of making a bigger seed is balanced by the resulting cost to seed number. Our results indicate flat cost corrected fitness of seed size as theory predicts.