Community trait assembly, the formation of distributions of phenotypic characteristics across coexisting species, can occur via two main processes: the filtering of trait distributions in the regional pool and in situ phenotypic evolution in local communities. But the relative importance of these processes remains unclear, largely because of the difficulty in determining the timing of evolutionary trait changes and biogeographic dispersal events in phylogenies. We assessed evolutionary and biogeographic transitions in woody plant species across the Indo-Malay Archipelago, a series of island groups where the same plant lineages interact with different seed disperser and seed predator assemblages. Fruit size in 2650 taxa spanning the angiosperm tree-of-life tended to be smaller in the Sulawesi and Maluku island groups, where frugivores are less diverse and smaller-bodied, than in the regional source pool. While numerous plant lineages (not just small-fruited ones) reached the isolated islands, colonists tended to be the smaller-fruited members of each clade. Nearly all evolutionary transitions to smaller fruit size predated, often substantially, organismal dispersal to the islands. Our results suggest that filtering rather than within-island evolution largely determined the distribution of fruit sizes in these regions.