Sexual size dimorphism results when female and male body size is influenced differently by natural and sexual selection. Typically, in polygynous species larger male body size is thought to be favored in competition for mates and constraints on maximal body size are due to countervailing natural selection on either sex; however, it has been postulated that sexual selection itself may result in stabilizing selection at an optimal mass. Here we test this hypothesis by retrospectively assessing the influence of body mass, one metric of body size, on the fitness of 113 wild-derived house mice (Mus musculus) residing within ten replicate semi-natural enclosures from previous studies conducted by our laboratory. Enclosures possess similar levels of sexual selection, but relaxed natural selection, relative to natural systems. Heavier females produced more offspring, while males of intermediate mass had the highest fitness. Female results suggest that some aspect of natural selection, absent from enclosures, acts to decrease their body mass, while the upper and lower boundaries of male mass are constrained by sexual selection.