Adaptive phenotypic plasticity is an important source of intraspecific variation, and for many plastic traits the costs or factors limiting plasticity seem cryptic. However, there are several different factors that may constrain the evolution of plasticity, but few models have considered costs and limiting factors simultaneously. Here we use a simulation model to investigate how the optimal level of plasticity in a population depends on a fixed fitness cost for maintaining a potential for plasticity or an incremental fitness cost for producing a plastic response in combination with environmental unpredictability (environmental fluctuation speed) limiting plasticity. Our model identifies two mechanisms that act, almost separately, to constrain the evolution of plasticity: 1) The fitness cost of plasticity scaled by the non-plastic environmental tolerance, and 2) the environmental fluctuation speed scaled by the rate of phenotypic change. That is, the evolution of plasticity is constrained by the high cost of plasticity in combination with high tolerance for environmental variation, or fast environmental changes in combination with slow plastic response. Qualitatively similar results are found when fixed and incremental fitness costs of plasticity are incorporated, although a larger degree of plasticity is selected for with an incremental cost. Our model highlights that it is important to consider direct fitness costs and phenotypic limitations in relation to non-plastic environmental tolerance and environmental fluctuations, respectively, to understand what constrains the evolution of phenotypic plasticity.