Historical abiotic factors such as climatic oscillations and extreme climatic events as well as biotic factors have shaped the structuring of species’ genetic diversity. In obligate species-specific mutualisms, the biogeographic histories of the interacting species are tightly linked. This could be particularly true for nuclear genes in the Ficus-pollinating wasp mutualistic association as the insects disperse pollen from their natal tree. In this study we compare spatial genetic structure of plant and pollinator for the Ficus hirta-Valisia javana association throughout South-East China including Hainan Island, for both nuclear and cytoplasmic markers. We show that dispersal of the insect leads to plant and insect presenting similar signatures of lack of genetic isolation by distance for nuclear genes on the continent over a distance of 1000 km. But we also show that the demographic histories of plant and insect are strikingly different. This is in agreement with extreme climatic events leading to transient regional extinctions of the insects, associated with local survival of the plants. We also evidence genetic differentiation for both wasps and fig-tree between the continent and Hainan Island, although the Qiongzhou Strait is only on average 30 km wide suggesting that geographic isolation by itself has not been sufficient to generate this differentiation. Hence, our results suggest that in highly dispersive mutualistic systems, isolation by dispersal limitation across a geographic barrier could be supplemented by isolation by adaptation, and maybe by coevolution, allowing further genetic divergence. In such systems, species may frequently be composed of a single population.