To cope with the increasing and less predictable temperature forecasts under climate change, many terrestrial ectotherms will have to migrate or rely on adaptation through plastic or evolutionary means. Studies suggest that some ectotherms have a limited potential to change their upper thermal limits via evolutionary shifts, but research has mostly focused on adult life stages under laboratory conditions. Here we use replicate populations of Drosophila melanogaster and a nested half sib/full sib quantitative genetic design to estimate heritabilities and genetic variance components for egg-to-adult viability under both laboratory and semi-natural field conditions, encompassing cold, benign and hot temperatures in two separate populations. The results demonstrated temperature-specific heritabilities and additive genetic variances for egg-to-adult viability. Heritabilities and genetic variances were higher under cold and benign compared to hot temperatures when tested under controlled laboratory conditions. Tendencies towards lower evolutionary potential at higher temperatures were also observed under semi-natural conditions although the results were less clear in the field setting. Overall the results suggest that ectotherms that already experience temperatures close to their upper thermal tolerance limits have a restricted capacity to adapt to higher temperatures by evolutionary means.