Sex differences in selection arise for at least two possible reasons: 1) differences originating from anisogamy – the Darwin-Bateman paradigm – and 2) competition-driven ecological character displacement (ECD), agnostic of anisogamy. Despite mounting evidence of ECD and increasing focus on the ecological causes and consequences of sexual dimorphism, progress in understanding the evolution of ecological sex differences has likely been hindered because ecological dimorphisms are not exclusive to ECD. I argue that embracing non-exclusivity of causal models of sexual dimorphism itself may provide insight into evolution of sex differences. This integrated view of the evolution of sexual dimorphism leads to four predictions for how sex-specific selection and phenotypic divergence between the sexes change over the course of the evolution of sexual dimorphism. First, dimorphism resulting directly from anisogamy likely precedes evolution of ecological dimorphism driven by ECD. Second, ecological sexual dimorphism driven by ECD may (initially) evolve in directions in trait space favored by other sources of sex-specific selection. Third, we may expect correlated evolution of ecological dimorphism and other forms of sexual dimorphism. Finally, ecological optima may be sex specific even when competition plays a role in reaching them. Rather than simply a less-parsimonious alternative explanation for ecological sex differences, ECD should be seen as one likely contributor to sex-specific selection that could act at predictable times during the evolution of ecological sexual dimorphisms.