Sexual selection can increase rates of adaptation by imposing stronger selection in males, thereby allowing efficient purging of the mutation load on population fitness at a low demographic cost. Indeed, sexual selection tends to be male-biased throughout the animal kingdom, but little empirical work has explored the ecological sensitivity of this sex difference. In this study, we generated theoretical predictions of sex-specific strengths of selection, environmental sensitivities and genotype-by-environment interactions, and tested them in seed beetles by manipulating either larval host plant or rearing temperature. Using fourteen isofemale lines, we measured sex-specific reductions in fitness components, genotype-by-environment interactions, and strength of selection (variance in fitness) in the juvenile and adult stage. As predicted, variance in fitness increased with stress, was consistently greater in males than females for adult reproductive success (implying strong sexual selection), but was similar in the sexes in terms of juvenile survival across all stress-levels. While genetic variance in fitness increased in magnitude under severe stress, heritability decreased, and particularly so in males. Moreover, genotype-by-environment interactions for fitness were common but specific to the type of stress and each sex and life stage, suggesting that new environments may change the relative alignment and strength of selection in males and females. Our study thus exemplifies how environmental stress can influence the relative forces of natural and sexual selection, as well as concomitant changes in genetic variance in fitness, which are predicted to have consequences for rates of adaptation in sexual populations.