In the seaweed fly, Coelopa frigida, a large chromosomal inversion system is affected by sexual selection and viability selection. However, our understanding of the interaction between these two selective forces is currently limited as research has focused upon a limited range of environments. We allowed C. frigida larvae to develop in two different algae, Fucus and Laminaria, and then measured viability and body size for each inversion genotype. Significant male-specific genotype-by-environment interactions influenced viability and body size. For males developing in Laminaria, the direction of viability selection acts similarly on the inversion system as the direction of sexual selection. In contrast, for males developing in Fucus, viability selection opposes sexual selection. These results demonstrate that through considering viability selection in different environments, the costs and benefits associated with sexual selection can be found to vary.