Understanding which environmental variables and traits underlie adaptation to harsh environments is difficult since many traits evolve simultaneously as populations or species diverge. Here we investigate the ecological variables and traits that underlie Mimulus laciniatus’ adaptation to granite outcrops compared to its sympatric, mesic-adapted progenitor M. guttatus. We use fine scale measurements of soil moisture and herbivory to examine differences in selective forces between the species’ habitats, and measure selection on flowering time, flower size, plant height, and leaf shape in a reciprocal transplant using M. laciniatus x M. guttatus F4 hybrids. We find that differences in drought & herbivory drive survival differences between habitats, that M. laciniatus and M. guttatus are each better adapted to their native habitat, and differential habitat selection on flowering time, plant stature, and leaf shape. While early flowering time, small stature, and lobed leaf shape underlie plant fitness in M. laciniatus’ seasonally dry environment, increased plant size is advantageous in a competitive mesic environment replete with herbivores like M. guttatus’. Given that we observed divergent selection between habitats in the direction of species differences, we conclude that adaptation to different microhabitats is an important component of reproductive isolation in this sympatric species pair.