Many specialist herbivores eat foods that are apparently low quality. The compensatory benefits of a poor diet may include protection from natural enemies. Several bee lineages specialize on pollen of the plant family Asteraceae, which is known to be a poor-quality food. Here we tested the hypothesis that specialization on Asteraceae pollen protects bees from parasitism. We compared rates of brood parasitism by Sapyga wasps on Asteraceae-specialist, Fabeae-specialist, and other species of Osmia bees in the field over several years and sites and found that Asteraceae-specialist species were parasitized significantly less frequently than other species. We then tested the effect of Asteraceae pollen on parasites by raising Sapyga larvae on three pollen mixtures: Asteraceae, Fabeae, and generalist (a mix of primarily non-Asteraceae pollens). Survival of parasite larvae was significantly reduced on Asteraceae provisions. Our results suggest that specialization on low-quality pollen may evolve because it helps protect bees from natural enemies.