Reductions in animal body size over recent decades are often interpreted as an adaptive evolutionary response to climate warming. However, for reductions in size to reflect adaptive evolution, directional selection on body size within populations must have become negative, or, where already negative, more so, as temperatures increased. To test this hypothesis, we performed a meta-analysis of the association between annual estimates of directional selection on body size from wild populations and annual mean temperatures in 39 studies. We found no evidence that warmer environments were associated with selection for smaller size. Instead, selection consistently favoured larger individuals, and was invariant to temperature. These patterns were similar in ectotherms and endotherms. An analysis using year rather than temperature revealed similar patterns, suggesting no evidence that selection has changed over time, and also indicating that the lack of association with annual temperature was not an artefact of choosing an erroneous time window for aggregating the temperature data. Although phenotypic trends in size will be driven by a combination of genetic and environmental factors, our results suggest little evidence for a necessary ingredient - negative directional selection - for declines in body size to be considered an adaptive evolutionary response to changing selection pressures.