Many natural populations exhibit temporal fluctuations in abundance that are consistent with external forcing by a randomly changing environment. As fitness emerges from an interaction between the phenotype and the environment, such demographic fluctuations probably include a substantial contribution from fluctuating phenotypic selection. We study the stochastic population dynamics of a population exposed to random (plus possibly directional) changes in the optimum phenotype for a quantitative trait that evolves in response to this moving optimum. We derive simple analytical predictions for the distribution of log population size over time both transiently and at stationarity under Gompertz density regulation. These predictions are well matched by population- and individual-based simulations. The log population size is approximately reverse gamma distributed, with a negative skew causing an excess of low relative to high population sizes, thus increasing extinction risk relative to a symmetric (e.g., normal) distribution with the same mean and variance. Our analysis reveals how the mean and variance of log population size change with the variance and autocorrelation of deviations of the evolving mean phenotype from the optimum. We apply our results to the analysis of evolutionary rescue in a stochastic environment and show that random fluctuations in the optimum can substantially increase extinction risk by both reducing the expected growth rate and increasing the variance of population size by several orders of magnitude.