Invasive exotic plants often impact native plant species through strong competition, yet their effects can vary across different native populations. Previous exposure to invasive species but also differentiation along abiotic gradients may determine to what extent native populations are competitively suppressed by plant invasive species. Here, we experimentally investigated whether competitive effects of the invasive exotic jewelweed Impatiens parviflora on its native congener Impatiens noli-tangere are related to previous exposure to the invader or rather to topography of the source populations, i.e., to terrain features such as elevation, aspect, or slope. We also asked whether, in return, topographic settings of invasive I. parviflora populations may explain competitive effects of the native jewelweed. In a common garden competition experiment we found that populations of native jewelweed from colder, higher altitude sites and from drier sites were on average seven and six times, respectively, more tolerant to interspecific competition than populations from warmer and wetter sites. In contrast, previous exposure of native jewelweed populations to the invader was not a significant predictor of invader’s competitive effects. Further, topographic settings of populations of invasive jewelweed were not significantly related to the native’s competitive effects. Our study suggests that the native jewelweed’s population differentiation along topographic gradients is more important in determining competitive interactions with an invasive congener than invasion-induced selection. As fine-scale environmental variation can shape adaptation of native species to the effects of invasive species, such variation should be considered when predicting invasive species’ impact on native plant communities.