Sexual size dimorphism (SSD) describes divergent body sizes of adult males and females. While SSD has traditionally been explained by sexual and fecundity selection, recent advances in physiology and developmental biology emphasize that SSD would occur proximately because of sexual differences in ontogenetic growth trajectories (i.e., growth rate and duration). Notably, these ontogenetic traits are subject to energetic or time constraints and thus traded off with fitness components (e.g., survival and reproduction). To elucidate the importance of such ontogenetic trade-offs in the evolution of SSD, we developed a new theoretical framework by extending quantitative genetic models for the evolution of sexual dimorphism in which we reinterpret the trait as body size and reformulate sex-specific fitness in size-dependent manners. More specifically, we assume that higher growth rate or longer growth duration leads to larger body size and higher reproductive success but incurs the cost of lower survivorship or shorter reproduction period. We illustrate how two sexes would optimize ontogenetic growth trajectories in sex-specific ways and exhibit divergent body sizes. The present framework provides new insights into the evolutionary theory of SSD and predictions for empirical testing.