Hybrid male sterility often evolves before female sterility or inviability of hybrids, implying that the accumulation of divergence between separated lineages should lead hybrid male sterility to have a more polygenic basis. However, experimental evidence is mixed. Here, we use the nematodes Caenorhabditis remanei and C. latens to characterize the underlying genetic basis of asymmetric hybrid male sterility and hybrid inviability. We demonstrate that hybrid male sterility is consistent with a simple genetic basis, involving a single X-autosome incompatibility. We also show that hybrid inviability involves more genomic compartments, involving diverse nuclear-nuclear incompatibilities, a mito-nuclear incompatibility, and maternal effects. These findings demonstrate that male sensitivity to genetic perturbation may be genetically simple compared to hybrid inviability in Caenorhabditis and motivates tests of generality for the genetic architecture of hybrid incompatibility across the breadth of phylogeny.