1. It is well documented that large herbivores have pronounced effects on plant communities in grassland ecosystems, and the extent and course of their effects can largely depend on both plant and herbivore characteristics. Previous studies highlighted the importance of plant productivity in predicting the impact of herbivores on grasslands. Yet, there has been little consideration of how different herbivores affect plant communities that, in turn, differ in plant diversity. 2. In a 2-year grazing experiment, we tested the effects of large herbivores (cattle or sheep, or both together) on plant communities under high and low plant diversity levels in eastern Eurasian steppe. 3. We found that, for high plant diversity grassland, mixed grazing by cattle and sheep significantly increases plant diversity, but we found no effect of grazing by cattle or sheep alone. Grazing by cattle or sheep alone or mixed grazing by cattle and sheep did not significantly affect plant biomass in the high diversity grassland. However, for low plant diversity grassland, grazing by cattle alone and mixed grazing by cattle and sheep significantly increased plant diversity, but significantly decreased plant biomass. There was no significant impact on both plant diversity and biomass from sheep grazing. 4. Synthesis and applications. We conclude that the effects of grazing in grassland strongly depend on herbivore assemblages and pre-grazing plant diversity. Herbivore grazing might contribute more to the maintenance of grassland structure and ecosystem functioning under high plant diversity compared with low plant diversity. Furthermore, our data suggest that multiple-species mixed grazing regimes in grassland systems with high plant diversity could represent the optimal protocol for grazing management. This study emphasizes the importance of maintaining both plant and herbivore diversity to optimize ecosystem functioning.