Larger populations generally adapt faster to their existing environment. However, it is unknown if the population size experienced during evolution influences the ability to face sudden environmental changes. To investigate this issue, we subjected replicate Escherichia coli populations of different sizes to experimental evolution in an environment containing a cocktail of three antibiotics. In this environment, the ability to actively efflux molecules outside the cell is expected to be a major fitness-affecting trait. We found that all the populations eventually reached similar fitness in the antibiotic cocktail despite adapting at different speeds, with the larger populations adapting faster. Surprisingly, whereas efflux activity enhanced in the smaller populations, it decayed in the larger ones. The evolution of efflux activity was largely shaped by pleiotropic responses to selection and not by drift. This demonstrates that quantitative differences in population size can lead to qualitative differences (decay/enhancement) in the fate of a character during adaptation to identical environments. Furthermore, the larger populations showed inferior fitness upon sudden exposure to several alternative stressful environments. These observations provide a novel link between population size and vulnerability to environmental changes. Counter-intuitively, adapting in larger numbers can render bacterial populations more vulnerable to abrupt environmental changes.