Sex differences in lifespan are ubiquitous across the tree of life and exhibit broad taxonomic patterns that remain a puzzle, such as males living longer than females in birds and vice versa in mammals. The prevailing “unguarded-X” hypothesis explains sex differences in lifespan by differential expression of recessive mutations on the X or Z chromosome of the heterogametic sex but has only received indirect support to date. An alternative hypothesis is that the accumulation of deleterious mutations and repetitive elements on the Y or W chromosome might lower the survival of the heterogametic sex (“toxic Y” hypothesis). Here, we use a new database to report lower survival of the heterogametic relative to the homogametic sex across 136 species of birds, mammals, reptiles and amphibians, as expected if sex chromosomes shape sex-specific lifespans, and consistent with previous findings. We also found that the relative sizes of both the X and the Y chromosomes in mammals (but not the Z or the W chromosomes in birds) are associated with sex differences in lifespan, as predicted by the “unguarded-X” and the “toxic Y”. Furthermore, we report that the relative size of the Y is negatively associated with male lifespan in mammals, so that small Y size correlates with increased male lifespan. In theory, toxic Y effects are expected to be particularly strong in mammals, and we did not find similar effects in birds. Our results confirm the role of sex chromosomes in explaining sex differences in lifespan across tetrapods and further suggest that, at least in mammals, “toxic Y” effects may play an important part in this role.