Sex-biased natal dispersal is widespread, and its significance remains a central question in evolutionary biology. However, theory so far fails to predict some of the most common patterns found in nature. To address this, we present novel results from an individual-based model investigating the joint roles of inbreeding load, demographic stochasticity, environmental stochasticity, and dispersal costs for the evolution of sex-biased dispersal. Most strikingly, we found that male-biased natal dispersal evolved in polygynous systems as a result of the interplay between inbreeding avoidance and stochasticity, whereas previous theory, in contrast to empirical observations, predicted male philopatry and female-biased natal dispersal under inbreeding load alone. Furthermore, the direction of the bias varied according to the nature of stochasticity. Our results therefore provide a unification of previous theory, yielding a much better qualitative match with empirical observations of male-biased dispersal in mate defense mating systems.