The margins of an expanding range are predicted to be challenging environments for adaptation. Marginal populations should often experience low effective population sizes (Ne) where genetic drift is high due to demographic expansion and/or census population size is low due to unfavorable environmental conditions. Nevertheless, invasive species demonstrate increasing evidence of rapid evolution and potential adaptation to novel environments encountered during colonization, calling into question whether significant reductions in Ne are realized during range expansions in nature. Here we report one of the first empirical tests of the joint effects of expansion dynamics and environment on effective population size variation during invasive range expansion. We estimate contemporary values of Ne using rates of linkage disequilibrium among genome-wide markers within introduced populations of the highly invasive plant Centaurea solstitialis (yellow starthistle) in North America (California, USA), and within native Eurasian populations. As predicted, we find that Ne within the invaded range is positively correlated with both expansion history (time since founding) and habitat quality (abiotic climate). History and climate had independent additive effects with similar effect sizes, indicating an important role for both factors in this invasion. These results support theoretical expectations for the population genetics of range expansion, though whether these processes can ultimately arrest the spread of an invasive species remains an unanswered question.