Local adaptation can be a fundamental component of speciation, but its dynamics in relation to gene flow are not necessarily straightforward. Herbivorous taxa with localized host plant or habitat specialization across their geographic range are ideal models for investigating the patterns and constraints of local adaptation and its impact on diversification. The charismatic, day-flying moths of the Hemileuca maia species complex (Lepidoptera: Saturniidae) are such taxa, as they are geographically-widespread, exhibit considerable ecological and morphological variability and host and habitat specificity, but apparently lack genetic differentiation across their range. Here, we use genome-wide single nucleotide polymorphisms to assess relationships and population structure of this group across North America, and investigate the scales where genomic divergence correlates with adaptive ecological characteristics. In contrast to previous genetic studies of the group, we find broad- and fine-scale genetic differentiation between lineages, which is at odds with various levels of taxonomic description and recognition of conservation units. Furthermore, ecological specialization only explains some fine-scale genetic differentiation, and across much of the group’s range, local adaptation is apparently occurring in the face of strong gene flow. These results provide unprecedented insight into drivers of speciation in this group, the relationship between taxonomy and genomics-informed species boundaries, and conservation management of internationally-protected entities. Broadly, this system provides a model for understanding how local adaptation in an herbivore can arise and be maintained in the face of apparently strong gene flow, and the importance of geographic isolation in generating genomic divergence, despite a lack of ecological divergence.