The role of divergent selection between alternative environments is well-recognized to promote reproductive isolation (RI) between lineages. However, most studies view each divergent environment as homogenous, thereby overlooking the potential role of within environment variation on RI between differentiating lineages. Here we test the importance of microenvironmental variation on RI using individual trees of two host plants each harboring locally adapted populations of the cynipid wasp, Belonocnema treatae.  We compared the fitness surrogate (survival) of offspring from hybrid crosses with residents across individuals of each  host plant Quercus virginiana and Q. geminata. We found evidence of weak hybrid inviability between host-associated lineages of B. treatae despite strong genomic differentiation. However, averaging across environments, masked great variation in hybrid fitness on individual trees, where hybrids performed worse than, equal to, or better than residents. Considering the environmental context of hybridization is thus critical to improving the predictability of divergence under variable selection.