Since most species are collections of genetically variable populations distributed to habitats differing in their abiotic/biotic environmental factors and community composition, the pattern and strength of natural selection imposed by species on each others' traits are also expected to be highly spatially variable. Here, we used genomic and quantitative genetic approaches to understand how spatially variable selection operates on the genetic basis of plant defenses to herbivores. To this end, an F2 progeny was generated by crossing Datura stramonium (Solanaceae) parents from two populations differing in their level of chemical defense. This F2 progeny was reciprocally transplanted into the parental plants’ habitats and by measuring the Identity by Descent (IBD) relationship of each F2 plant to each parent, we were able to elucidate how spatially variable selection imposed by herbivores operated on the genetic background (IBD) of resistance to herbivory, promoting local adaptation. The results highlight that plants possessing the highest total alkaloid concentrations (sum of all alkaloid classes) were not the most well-defended or fit. Instead, specific alkaloids and their linked loci/alleles were favored by selection imposed by different herbivores. This has led to population differentiation in plant defenses and thus, to local adaptation driven by plant-herbivore interactions.