The seasonal match between folivore and leaf phenology affects the annual success of arboreal folivore populations because many folivores exploit developing leaves, which are an ephemeral resource. One strategy for folivores to exploit early-season leaves is to anticipate their emergence. The consequence of this behavior for trees is that individuals that set leaves earlier may experience greater rates of folivore damage, with potential negative fitness consequences. To test this hypothesis, we surveyed the early-season phenology, insect folivore damage, and acorn crop of a population of valley oaks (Quercus lobata) over a 3-year period. We found that trees that set leaves earlier experienced greater rates of folivore damage than trees that set leaves later in the season. In addition, we observed a lagged effect of folivore damage on acorn production, whereby trees with greater leaf damage produced fewer acorns in the subsequent year. These results indicate potential negative fitness consequences of earlier leaf phenology. Our study suggests that folivore pressure may be one factor that affects the optimal timing of leaf set in oaks.