Humans exhibit a high level of vocal plasticity in speech production which allows us to acquire both native and foreign languages, dialects and adapt to local accents in social communication. In comparison, non-human primates exhibit limited vocal plasticity, especially in adulthood, which would limit their ability to adapt to different social and environmental contexts in vocal communication. Here we quantitatively examined the ability of adult common marmosets (Callithrix jacchus), a highly vocal New World primate species, to modulate their vocal production in social contexts. While recent studies have demonstrated vocal learning in developing marmosets, we know much less on the extent of vocal learning and plasticity in adult marmosets. We found in the present study that marmosets were able to adaptively modify the spectrotemporal structure of their vocalizations when they encountered interfering sounds. Our experiments showed that marmosets shifted the spectrum of their vocalizations away from the spectrum of the interfering sounds in order to avoid the overlap. More interestingly, we found that marmosets made predictive and long-lasting spectral shifts in their vocalizations after they had experienced a particular type of interfering sounds. These observations provided evidence for directional control of the vocalization spectrum and long-term vocal plasticity by adult marmosets. Findings reported here have important implications for the ability of this New World primate species in voluntarily and adaptively controlling their vocal production in social communication.