While extensive research has focused on how social interactions evolve, the fitness consequences of the neuroendocrine mechanisms underlying these interactions have rarely been documented, especially in the wild. Here, we measure how the neuroendocrine mechanisms underlying male behavior affecting mating success and sperm competition in the ocellated wrasse (Symphodus ocellatus). In this species, males exhibit three alternative reproductive types. ‘Nesting males’ provide parental care, defend territories, and form cooperative associations with unrelated ‘satellites’, who cheat by sneaking fertilizations but help by reducing sperm competition from ‘sneakers’ who do not cooperate or provide care. To measure the fitness consequences of the mechanisms underlying these social interactions, we used “phenotypic engineering” that involved administering an androgen receptor antagonist (flutamide) to wild, free-living fish. Nesting males treated with flutamide shifted their aggression from sneakers to satellite males and experienced decreased submissiveness by sneaker males (which correlated with decreased nesting male mating success). The preoptic area (POA), a region controlling male reproductive behaviors, exhibited dramatic down-regulation of androgen receptor (AR) and vasotocin 1a receptor (V1aR) mRNA following experimental manipulation of androgen signaling. We did not find a direct effect of the manipulation on male mating success, paternity or larval production. However, variation in neuroendocrine mechanisms generated by the experimental manipulation was significantly correlated with changes in behavior and mating success: V1aR expression was negatively correlated with satellite-directed aggression and expression of its ligand arginine vasotocin (AVT) was positively correlated with courtship and mating success, thus revealing the potential for sexual selection on these mechanisms.