Natural selection by visually-hunting predators has led to evolution of color defense strategies such as masquerade, crypsis and aposematism that reduce the risk of predation in prey species. These color defenses are not mutually exclusive, and switches between strategies with ontogenic development are widespread across taxa. However, the evolutionary dynamics of ontogenic color change are poorly understood. Using comparative phylogenetics, we studied the evolution of color defenses in the complex lifecycles of swallowtail butterflies (Family: Papilionidae). We also tested the relative importance of life history traits, chemical and visual backgrounds, and ancestry on the evolution of protective coloration. We found that vulnerable early and late instar caterpillars of species that feed on sparsely-vegetated, toxic plants were aposematic, whereas species that feed on densely-vegetated, non-toxic plants had masquerading and cryptic caterpillars. Masquerading caterpillars resembled bird droppings at early instars, and transitioned to crypsis with an increase in body size at late instars. The immobile pupae – safe from motion-detecting, visually hunting predators – retained the ancestral cryptic coloration in all lineages irrespective of the toxic nature of the host plant. Thus, color defense strategy (masquerade, crypsis or aposematism) at a particular lifestage in the lifecycle of swallowtail butterflies was determined by the interaction between life history traits such as body size and motion levels, phytochemical and visual backgrounds, and ancestry. We show that ontogenic color change in swallowtail butterflies is an adaptive response to age-dependent vulnerability to predation.