The fitness of hybrids might be compromised as a result of intrinsic isolation and/or because they fall between ecological niches due to their intermediate phenotypes (“extrinsic isolation”). Here, we present data from several crosses (parental crosses, F1, F2 and backcrosses) between the two host races of Lochmaea capreae on willow and birch to test for extrinsic isolation, intrinsic isolation, and environmentally dependent genetic incompatibilities. We employed a reciprocal transplant design in which offspring were raised on either host plant and their survival was recorded until adulthood. We also applied joint-scaling analysis to determine the genetic architecture of hybrid inviability. The relative fitness of the backcrosses switched between environments; furthermore, the additive genetic-environment interaction was detected as the strongest effect in our analysis. These results provide strong evidence that divergent natural selection has played a central role in the evolution of hybrid dysfunction between host races. Joint-scaling analysis detected significant negative epistatic effects which are most evident in the poor performance of F2-hybrids on willow, indicating signs of intrinsic isolation. We did not find any evidence that genetic incompatibilities are manifested independently of environmental conditions. Our findings suggest the outcome of natural hybridization between these host races is mainly affected by extrinsic isolation and a weak contribution of intrinsic isolation.