Leaf phenology is important to herbivores, but the timing and extent of leaf drop has not played an important role in our understanding of herbivore interactions with deciduous plants. Using phylogenetic general least squares regression, we compared the phenology of leaves of 55 oak species in a common garden with the abundance of leaf miners on those trees. Mine abundance was highest on trees with an intermediate leaf retention index, i.e. trees that lost most, but not all, of their leaves for 2–3 months. The leaves of more evergreen species were more heavily sclerotized, and sclerotized leaves accumulated fewer mines in the summer. Leaves of more deciduous species also accumulated fewer mines in the summer, and this was consistent with the idea that trees reduce overwintering herbivores by shedding leaves. Trees with a later leaf set and slower leaf maturation accumulated fewer herbivores. We propose that both leaf drop and early leaf phenology strongly affect herbivore abundance and select for differences in plant defense. Leaf drop may allow trees to dispose of their herbivores so that the herbivores must recolonize in spring, but trees with the longest leaf retention also have the greatest direct defenses against herbivores.