Determining the processes responsible for phenotypic variation is one of the central tasks of evolutionary biology. While the importance of acoustic traits for foraging and communication in echolocating mammals suggests adaptation, the seldom-tested null hypothesis to explain trait divergence is genetic drift. Here we derive FST values from multi-locus coalescent isolation-with-migration models, and couple them with estimates of quantitative trait divergence, or PST, to test drift as the evolutionary process responsible for phenotypic divergence in island populations of the Pteronotus parnellii species complex. Compared to traditional comparisons of PST to FST, the migration-based estimates of FST are unidirectional instead of bidirectional, simultaneously integrate variation among loci and individuals, and posterior densities of PST and FST can be compared directly. We found the evolution of higher call frequencies is inconsistent with genetic drift for the Hispaniolan population, despite many generations of isolation from its Puerto Rican counterpart. While the Hispaniolan population displays dimorphism in call frequencies, the higher frequency of the females is incompatible with sexual selection. Instead, cultural drift toward higher frequencies among Hispaniolan females might explain the divergence. By integrating Bayesian coalescent and trait analyses, this study demonstrates a powerful approach to testing genetic drift as the default evolutionary mechanism of trait differentiation between populations.