Trait variation is normally separated into genetic and environmental components, yet genetic factors also control the expression of environmental variation, encompassing plasticity across environmental gradients and within-environment responses. We defined four components of environmental variation: plasticity across environments, variability in plasticity, variation within environments, and differences in within-environment variation across environments. We assessed these components for cold tolerance across five rearing temperatures using the Drosophila melanogaster Genetic Reference Panel (DGRP). The four components were found to be heritable, and genetically correlated to different extents. By whole genome single marker regression, we detected multiple candidate genes controlling the four components and showed limited overlap in genes affecting them. Using the binary UAS-GAL4 system, we functionally validated the effects of a subset of candidate genes affecting each of the four components of environmental variation and also confirmed the genetic and phenotypic correlations obtained from the DGRP in distinct genetic backgrounds. We delineate selection targets associated with environmental variation and the constraints acting upon them, providing a framework for evolutionary and applied studies on environmental sensitivity. Based on our results we suggest that the traditional quantitative genetic view of environmental variation and genotype-by-environment interactions needs revisiting.