Evolutionary biologists have often assumed that ecological generalism comes at the expense of less intense exploitation of specific resources and that this trade-off will promote the evolution of ecologically specialized daughter species. Using a phylogenetic comparative approach with butterflies as a model system, we test hypotheses that incorporate changes in niche breadth and location into explanations of the taxonomic diversification of insect herbivores. Specifically, we compare the oscillation hypothesis, where speciation is driven by host-plant generalists giving rise to specialist daughter species, to the musical chairs hypothesis, where speciation is driven by host-plant switching, without changes in niche breadth. Contrary to the predictions of the oscillation hypothesis, we recover a negative relationship between host-plant breadth and diversification rate and find that changes in host breadth are seldom coupled to speciation events. By contrast, we present evidence for a positive relationship between rates of host switching and butterfly diversification, consonant with the musical chairs hypothesis. These results suggest that the costs of trophic generalism in plant-feeding insects may have been overvalued and that transitions from generalists to ecological specialists may not be an important driver of speciation in general.