Herbivory has long been recognized as a significant driver of plant population dynamics, yet its effects along environmental gradients are unclear. Understanding how weather modulates plant–insect interactions can be particularly important for predicting the consequences of exotic insect invasions, and an explicit consideration of weather may help explain why the impact can vary greatly across space and time. We surveyed two native prickly pear cactus species (genus Opuntia) in the Florida panhandle, USA, and their specialist insect herbivores (the invasive South American cactus moth, Cactoblastis cactorum, and three native insect species) for five years across six sites. We used generalized linear mixed models to assess the impact of herbivory and weather on plant relative growth rate (RGR) and sexual reproduction, and we used Fisher's exact test to estimate the impact of herbivory on survival. Weather variables (precipitation and temperature) were consistently significant predictors of vital rate variation for both cactus species, in contrast to the limited and varied impacts of insect herbivory. Weather only significantly influenced the impact of herbivory on Opuntia humifusa fruit production. The relationships of RGR and fruit production with precipitation suggest that precipitation serves as a cue in determining the trade-off in the allocation of resources to growth or fruit production. The presence of the native bug explained vital rate variation for both cactus species, whereas the invasive moth explained variation only for O. stricta. Despite the inconsistent effect of herbivory across vital rates and cactus species, almost half of O. stricta plants declined in size, and the invasive insect negatively affected RGR and fruit production. Given that fruit production was strongly size-dependent, this suggests that O. stricta populations at the locations surveyed are transitioning to a size distribution of predominantly smaller sizes and with reduced sexual reproduction potential.