The causes and consequences of fluctuating population densities are an important topic in ecological literature. Yet, the effects of such fluctuations on maintenance of variation in spatially structured populations have received little analytic treatment. We analyze what happens when two habitats coupled by migration not only differ in their trade-offs in selection but also in their demographic stability -- and show that equilibrium allele frequencies can change significantly due to ecological feedback arising from locally fluctuating population sizes. When an ecological niche exhibits such fluctuations, these drive an asymmetry in the relative impact of gene flow, and therefore, the equilibrium frequency of the locally adapted type decreases. Our results extend the classic conditions on maintenance of diversity under selection and migration by including the effect of fluctuating population densities. We find simple analytic conditions in terms of the strength of selection, immigration, and the extent of fluctuations between generations in a continent-island model. While weak fluctuations hardly affect coexistence, strong recurrent fluctuations lead to extinction of the type better adapted to the fluctuating niche -- even if the invader is locally maladapted. There is a disadvantage to specialization to an unstable habitat, as it makes the population vulnerable to swamping from more stable habitats.