Biological invasions are one of the major threats to biodiversity worldwide and contribute to changing community patterns and ecosystem processes. However, it is often not obvious whether an invader is the “driver” causing ecosystem changes or a “passenger” which is facilitated by previous ecosystem changes. Causality of the impact can be demonstrated by experimental removal of the invader or introduction into a native community. Using such an experimental approach, we tested whether the impact of the invasive plant Impatiens glandulifera on native vegetation is causal, and whether the impact is habitat-dependent. We conducted a field study comparing invaded and uninvaded plots with plots from which I. glandulifera was removed and plots where I. glandulifera was planted within two riparian habitats, alder forests and meadows. A negative impact of planting I. glandulifera and a concurrent positive effect of removal on the native vegetation indicated a causal effect of I. glandulifera on total native biomass and growth of Urtica dioica. Species α-diversity and composition were not affected by I. glandulifera manipulations. Thus, I. glandulifera had a causal but low effect on the native vegetation. The impact depended slightly on habitat as only the effect of I. glandulifera planting on total biomass was slightly stronger in alder forests than meadows. We suggest that I. glandulifera is a “back-seat driver” of changes, which is facilitated by previous ecosystem changes but is also a driver of further changes. Small restrictions of growth of the planted I. glandulifera and general association of I. glandulifera with disturbances indicate characteristics of a back-seat driver. For management of I. glandulifera populations this requires habitat restoration along with removal of the invader.