Fencing is an effective and practical method for restoring degraded grasslands in northern China. However, little is known about the role of excess litter accumulation due to long-term fencing in regulating abiotic environment and driving changes in community structure and function. We conducted a three-year field experiment in two fenced grasslands in Inner Mongolia, and monitored light quantity, soil temperature, and soil moisture continuously, and determined community height, community aboveground net primary productivity (ANPP), and the relative dominance of different plant functional groups. Litter accumulation reduced light quantity and soil temperature but increased soil moisture. The regulating effects of litter accumulation on soil temperature and soil moisture fluctuated temporally and gradually weakened over the growing season. Litter accumulation also altered community vertical structure and function by increasing community height and ANPP. The increase in soil moisture increased the relative dominance of rhizome grasses but suppressed bunch grasses, thereby shifting bunch grass grasslands to rhizome grass grasslands. Our findings provide a potential mechanism for community succession in the context of litter accumulation in fenced grasslands, and indicate that the vegetation and ecosystem services of degraded grasslands are improved after appropriate fencing.