Whether sexual selection impedes or aids adaptation has become an outstanding question in times of rapid environmental change and parallels the debate about how the evolution of individual traits impacts on population dynamics. The net effect of sexual selection on population viability results from a balance between genetic benefits of “good genes” effects and costs of sexual conflict. Depending on how these facets of sexual selection are affected under environmental change, extinction of maladapted populations could either be avoided or accelerated. Here, we evolved seed beetles under three alternative mating regimes to disentangle the contributions of sexual selection, fecundity selection and male-female coevolution to individual reproductive success and population fitness. We compared these contributions between the ancestral environment and two stressful environments (elevated temperature and a host plant shift). We found evidence that sexual selection on males had positive genetic effects on female fitness components across environments, supporting good genes sexual selection. Interestingly, however, when males evolved under sexual selection with fecundity selection removed, they became more robust to both temperature and host plant stress compared to their conspecific females and males from the other evolution regimes that applied fecundity selection. We quantified the population-level consequences of this sex-specific adaptation and found evidence that the cost of socio-sexual interactions in terms of reduced offspring production was higher in the regime applying only sexual selection to males. Moreover, the cost tended to be more pronounced at the elevated temperature to which males from the regime were more robust compared to their conspecific females. These results illustrate the tension between individual-level adaptation and population-level viability in sexually reproducing species and suggest that the relative efficacies of sexual selection and fecundity selection can cause inherent sex-differences in environmental robustness that may impact demography of maladapted populations.