How the balance between selection, migration and drift influences the evolution of local adaptation has been under intense theoretical scrutiny. Yet, empirical studies that relate estimates of local adaptation to quantification of gene flow and effective population sizes have been rare. Here, we conducted a reciprocal transplant trial, a common garden trial, and a whole-genome based demography analysis to examine these effects among Arabidopsis lyrata populations from two altitudinal gradients in Norway. Demography simulations indicated that populations within the two gradients are connected by gene flow (0.1 < 4Nem < 11) and have small effective population sizes (Ne < 6,000), suggesting that both migration and drift can counteract local selection. However, the three-year field experiments showed evidence of local adaptation at the level of hierarchical multi-year fitness, attesting to the strength of differential selection. In the lowland habitat, local superiority was associated with greater fecundity, while viability accounted for fitness differences in the alpine habitat. We also demonstrate that flowering time differentiation has contributed to adaptive divergence between these locally adapted populations. Our results show that despite the estimated potential of gene flow and drift to hinder differentiation, selection among these A. lyrata populations has resulted in local adaptation.