Flowers and fruits are two major phases of plant reproduction that often use colorful signals to attract animal mutualists. Fruits develop from the ovaries of flowers, and both organs use the same suites of pigments to create color. Due to these developmental links and functional similarities, we sought to test for correlations in flower and fruit color lability across clades. We selected 51 clades (2960 species) of animal-pollinated and animal-dispersed (i.e., fleshy-fruited) plants and scored flower and fruit color into eight discrete (human-perceived) categories for the same set of species in each clade. We used stochastic character mapping to estimate the number and rates of transitions among colors in flowers and fruits. We demonstrate a negative correlation in the number of transitions in flower and fruit color across clades (R² = 0.41; p < 0.001). Among animal-pollinated and animal-dispersed clades, 67% are “fruit clades” biased towards fruit color lability, while 29% are “flower clades” biased towards flower color lability. Furthermore, clades with yellow- or orange-flowered species also tend to have those colors in their fruits, and red flowers are more common in “flower clades” and brown fruits in “fruit clades”. These patterns suggest that clades specialize on one phase of reproduction or the other. Possible explanations include constraints on energetic investment into either pollination or dispersal; environmental factors that select for color diversification in one organ but not the other; or constraints imposed by the underlying structure of pigment pathways.