Insect herbivores are important mediators of selection on traits that impact plant defense against herbivory and competitive ability. Although recent experiments demonstrate a central role for herbivory in driving rapid evolution of defense and competition-mediating traits, whether and how herbivory shapes heritable variation in these traits remains poorly understood. Here, we evaluate the structure and evolutionary stability of the G matrix for plant metabolites that are involved in defense and allelopathy in the tall goldenrod, Solidago altissima. We show that G has evolutionarily diverged between experimentally-replicated populations that evolved in the presence versus the absence of ambient herbivory, providing direct evidence for the evolution of G by natural selection. Specifically, evolution in an herbivore-free habitat altered the orientation of G, revealing a negative genetic covariation between defense- and competition-related metabolites that is typically masked in herbivore-exposed populations. Our results may be explained by predictions of classical quantitative genetic theory, as well as the theory of acquisition-allocation trade-offs. The study provides compelling evidence that herbivory drives the evolution of plant genetic architecture.