Understanding what limits or facilitates species' responses to human-induced habitat change can provide insight for the control of invasive species and the conservation of small populations, as well as an arena for studying adaptation to realistic novel environments. Small effective size of ancestral populations could limit the establishment in, or response to, a novel or altered habitat because of low genetic variation for ecologically important traits, and/or because small populations harbor fixed deleterious mutations. I estimated the fitness of individuals from populations of the endangered plant Hypericum cumulicola, of known census and effective size, transplanted into native scrub habitat and unpaved roadsides, which are a novel habitat for this species. I found a significant positive relationship between estimates of population size and mean fitness, but only in the novel roadside habitat. Fitness was more than 200% greater in the roadside habitat than the scrub, mostly due to increased fecundity. These results combined with previous estimates of heterosis in this species suggest that fixed deleterious mutations could contribute to lower fitness of field transplants from small populations in the novel environment.