Models of speciation-with-gene-flow have shown that the reduction in recombination between alternative chromosome arrangements can facilitate the fixation of locally adaptive genes in the face of gene flow and contribute to speciation. However, it has proven frustratingly difficult to show empirically that inversions have reduced gene flow and arose during or shortly after the onset of species divergence rather than represent ancestral polymorphisms. Here we present an analysis of whole genome data from a pair of cactophilic fruit flies, Drosophila mojavensis and D. arizonae, which are reproductively isolated in the wild and differ by several large inversions on three chromosomes. We found an increase in divergence at rearranged compared to colinear chromosomes. Using the density of divergent sites in short sequence blocks we fit a series of explicit models of species divergence in which gene flow is restricted to an initial period after divergence and may differ between colinear and rearranged parts of the genome. These analyses show that D. mojavensis and D. arizonae have experienced post-divergence gene flow which ceased around 270 KY ago and was significantly reduced in chromosomes with fixed inversions. Moreover, we show that these inversions most likely originated around the time of of species divergence which is compatible with theoretical models of speciation with gene flow.