Mating behaviours are diverse and noteworthy, especially within species radiations where they may contribute to speciation. Studying how differences in mating behaviours arise between species can help us understand how diversity is generated at multiple biological levels. The bioluminescent courtship displays of cypridinid ostracods (or sea fireflies) are an excellent system for this since amazing variety evolves while using a conserved biochemical mechanism. We find that the evolution of one aspect in this behavioural phenotype - the duration of bioluminescent courtship pulses - is shaped by biochemical function. First, by measuring light production from induced bioluminescence in 38 species, we discovered differences between species in their biochemical reactions. Then, for 16 species of which biochemical, phylogenetic, and behavioral data are all available, we used phylogenetic comparative models to show that differences in biochemical reaction are nonlinearly correlated with the duration of courtship pulses. This relationship indicates that changes to both enzyme (c-luciferase) function and usage have shaped the evolution of courtship displays, but that they differentially contribute to these phenotypic changes. This nonlinear dynamic may have consequences for the disparity of signaling phenotypes observed across species, and demonstrates how unappreciated diversity at the biochemical level can lead to inferences about behavioural evolution.