The fig and pollinator wasp obligate mutualism is diverse (~750 described species), ecologically important, and ancient (~80 Ma). Once thought to be an example of strict one-to-one cospeciation, current thinking suggests genera of pollinator wasps codiversify with corresponding sections of figs, but the degree to which cospeciation or other processes contribute to the association at finer scales is unclear. Here we use genome-wide sequence data from a community of Panamanian strangler figs and associated wasp pollinators to estimate the relative contributions of four evolutionary processes generating cophylogenetic patterns in this mutualism: cospeciation, host switching, pollinator speciation, and pollinator extinction. Using a model-based approach adapted from the study of gene family evolution, our results demonstrate the importance of host switching of pollinator wasps at this fine phylogenetic and regional scale. Although we estimate a modest amount of cospeciation, simulations reveal the number of putative cospeciation events to be consistent with what would be expected by chance. Additionally, model selection tests identify host switching as a critical parameter for explaining cophylogenetic patterns in this system. Our study demonstrates a promising approach through which the history of evolutionary association between interacting lineages can be rigorously modeled and tested in a probabilistic phylogenetic framework.