Sexual selection and ecological differences are important drivers of speciation. Much research has focused on female choice, yet the role of male competition in ecological speciation has been understudied. Here, we test how mating habitats impact sexual selection and speciation through male competition. Using limnetic and benthic species of threespine stickleback fish, we find that different mating habitats select differently on male traits through male competition. In mixed habitat with both vegetated and open areas, selection favors two trait combinations of male body size and nuptial color: large with little color and small with lots of color. This matches what we see in reproductively isolated stickleback species, suggesting male competition could promote trait divergence and reproductive isolation. In contrast, when only open habitat exists selection favors one trait combination, large with lots of color, which would hinder trait divergence and reproductive isolation. Other behavioral mechanisms in male competition that might promote divergence, such as avoiding aggression with heterospecifics, are insufficient to maintain separate species. This work highlights the importance of mating habitats in male competition for both sexual selection and speciation.