In flowering plants, shifts from outcrossing to partial or complete self-fertilization have occurred independently thousands of times, yet the underlying adaptive processes are difficult to discern. Selfing’s ability to provide reproductive assurance when pollination is uncertain is an oft-cited ecological explanation for its evolution, but this benefit may be outweighed by genetic costs diminishing its selective advantage over outcrossing. We directly studied the fitness effects of a self-compatibility (SC) mutation that was backcrossed into a self-incompatible (SI) population of Leavenworthia alabamica, illuminating the direction and magnitude of selection on the mating-system modifier. In array experiments conducted in two years, SC plants produced 17-26% more seed, but this advantage was counteracted by extensive seed discounting -- the replacement of high-quality outcrossed seeds by selfed seeds. Using a simple model and simulations, we demonstrate that SC mutations with these attributes rarely spread to high frequency in natural populations, unless inbreeding depression falls below a threshold value (0.57 ≤ threshold ≤ 0.70) in SI populations. A combination of heavy seed discounting and moderate inbreeding depression likely explains why outcrossing adaptations such as self-incompatibility are maintained generally, despite persistent input of selfing mutations and frequent limits on outcross seed production in nature.