Specialized signals emitted by competing males often convey honest information about fighting ability. It is generally believed that receivers use these signals to directly assess their opponents. Here, we demonstrate an alternative communication strategy used by males in a breeding system where the costs of conflict are extreme. We evaluated the acoustic displays of breeding male northern elephant seals (Mirounga angustirostris), and found that social knowledge gained through prior experience with signallers was sufficient to maintain structured dominance relationships. Using sound analysis and playback experiments with both natural and modified signals, we determined that males do not rely on encoded information about size or dominance status, but rather learn to recognize individual acoustic signatures produced by their rivals. Further, we show that behavioural responses to competitors' calls are modulated by relative position in the hierarchy: the highest ranking (alpha) males defend their harems from all opponents, whereas mid-ranking (beta) males respond differentially to familiar challengers based on the outcome of previous competitive interactions. Our findings demonstrate that social knowledge of rivals alone can regulate dominance relationships among competing males within large, spatially dynamic social groups, and illustrate the importance of combining descriptive and experimental methods when deciphering the biological relevance of animal signals.