Despite the fundamental importance of mutation to the evolutionary process, we have little knowledge of the direct consequences of specific spontaneous mutations to the fitness of the organism. Combining results of whole-genome sequencing with repeated field assays of survival and reproduction, we quantify the combined effects on fitness of spontaneous mutations identified in Arabidopsis thaliana. We demonstrate that the effects are beneficial, deleterious or neutral depending on the environmental context. Some lines, bearing mutations disrupting known loci, differ strongly in fitness from the founder or premutation genotype. Those effects vary across environments, e.g., a line with a major deletion spanning a transcription factor gene expressed lower fitness than the founder under most conditions but exceeded the founder’s fitness in one environment. The large contribution of genotype by environment interaction (GxE) to mutation effects on fitness implies spatial and/or temporal variation in selection on new mutations and could contribute to the maintenance of standing genetic variation.