Accurate age estimates underpin our everyday social interactions, the provision of age-restricted services and police investigations. Previous work suggests that these judgments are error-prone, but the processes giving rise to these errors are not understood. Here we present the first systematic test of bias in age estimation using a large database of standardized passport images of heterogeneous ages (n = 3948). In three experiments we tested a range of perceiver age groups (n = 84), and found average age estimation error to be approximately 8 years. We show that this error can be attributed to two separable sources of bias. First, and accounting for the vast majority of variance, our results show an assimilative serial dependency whereby estimates are systematically biased towards the age of the preceding face. Second, younger faces are generally perceived to be older than they are, and older faces to be younger. In combination, these biases account for around 95% of variance in age estimates. We conclude that that perception of age is modulated by representations that encode both a viewer’s recent and normative exposure to faces. The finding that age perception is subject to strong top-down influences based on our immediate experience has implications for our understanding of perceptual processes involved in face perception, and for improving accuracy of age estimation in important real world tasks.