During local adaptation with gene flow, some regions of the genome are inherently more responsive to selection than others. Recent theory predicts that X-linked genes should disproportionately contribute to local adaptation relative to other genomic regions, yet this prediction remains to be tested. We carried out a multi-generation crossing scheme, using two cline-end populations of Drosophila melanogaster, to estimate the relative contributions of the X chromosome, autosomes and mitochondrial genome to adaptive divergence in four traits involved in local adaptation (wing size, and resistance to heat, desiccation, and starvation stresses). We found that the mitochondrial genome and autosomes contributed significantly to clinal divergence in three of the four traits. In contrast, the X made no significant contribution to divergence in these traits. Given the small size of the mitochondrial genome, our results indicate that it plays a surprisingly large role in clinal adaptation. In contrast, the X, which represents roughly 20% of the Drosophila genome, contributes negligibly – a pattern that conflicts with theoretical predictions. These patterns reinforce recent work implying a central role of mitochondria in climatic adaptation, and suggest that different genomic regions may play fundamentally different roles in processes of divergence with gene flow.