Non-genetic inheritance media, from methyl-accepting cytosines to culture, tend to 'mutate' more frequently than DNA sequences. Whether or not this makes them inexhaustible suppliers for adaptive evolution will depend on the effect of non-genetic mutations (hereafter epimutations) on fitness-related traits. Here we investigate how the magnitude of these effects might themselves evolve, specifically whether natural selection may set boundaries to the adaptive potential of non-genetic inheritance media due to their higher mutability. In our model, the genetic and epigenetic contributions to a non-neutral phenotype are controlled by an epistatic modifier locus, which evolves under the combined effects of drift and selection. We show that a pure genetic control evolves when the environment is stable, provided that the population is large, such that the phenotype becomes robust to frequent epimutations. When the environment fluctuates, however, selection on the modifier locus also fluctuates and can overall produce a large non-genetic contribution to the phenotype, especially when the epimutation rate matches the rate of environmental variation. We further show that selection on the modifier locus is generally insensitive to recombination, meaning it is mostly direct, i.e. not relying on subsequent effects in future generations. These results suggest that unstable inheritance media might significantly contribute to fitness variation of traits subject to highly variable selective pressures, but little to traits responding to scarcely variable aspects of the environment. More generally, our study demonstrates that the rate of mutation and the adaptive potential of any inheritance media should not be seen as independent properties.