Persistence and adaptation in novel environments are limited by small population size, strong selection, and maladaptive gene flow. Mating system plasticity is common in angiosperms and may provide both demographic and genetic benefits that promote niche evolution, including reproductive assurance and isolation from maladaptive gene flow. Yet increased self-fertilization may also cause inbreeding depression, accumulation of deleterious mutations, and reduced adaptive potential. Here we use individual-based simulations to examine the consequences of mating system plasticity for persistence and adaptation in a novel environment that imposes selection on a quantitative trait. We examine the joint evolution of local adaptation, inbreeding depression, and genetic load. We find that a plastic shift to a mixed mating system generally promotes niche evolution by decreasing the risk of extinction, providing isolation from maladaptive gene flow, and temporarily increasing genetic variance in the trait under selection, whereas obligate self-fertilization reduces adaptive potential. These effects are most pronounced under conditions of mate limitation, strong selection, or maladaptive gene flow. Our results highlight the diverse demographic and genetic consequences of self-fertilization and support the potential role for plastic shifts in mating system to promote niche evolution in flowering plants.