Different modes of non-genetic inheritance are expected to affect population persistence in fluctuating environments. We here analyze Caenorhabiditis elegans density-independent per capita growth rate time series on 36 populations experiencing 6 controlled sequences of challenging oxygen level fluctuations across 60 generations, and parameterize competing models of non-genetic inheritance in order to explain observed dynamics. Our analysis shows that phenotypic plasticity and anticipatory maternal effects are sufficient to explain growth rate dynamics, but that a carryover model where ``epigenetic" memory is imperfectly transmitted and might be reset at each generation is a better fit to the data. We further find that this epigenetic memory is asymmetric since it is kept for longer when populations are exposed to the more challenging environment. Our analysis suggests that population persistence in fluctuating environments depends on the non-genetic inheritance of phenotypes whose expression is regulated across multiple generations.