In plant-arthropod associations, the first herbivores to colonise a plant may directly or indirectly affect community assembly on that particular plant. Whether the order of arrival of different arthropod species further modulates community assembly and affects plant fitness remains unclear. Using wild Brassica oleracea plants in the field, we manipulated the order of arrival of early-season herbivores that belong to different feeding guilds, namely the aphid Brevicoryne brassicae and caterpillars of Plutella xylostella. We investigated the effect of herbivore identity and order of arrival on community assembly on two B. oleracea plant populations during two growth seasons. For this perennial plant, we evaluated whether foliar herbivory also affected herbivore communities on the flowers and if these interactions affected plant seed production. Aphid infestation caused an increase in parasitoid abundance, but caterpillars modulated these effects, depending on the order of herbivore infestation and plant population. In the second growth season, when plants flowered, the order of infestation of leaves with aphids and caterpillars more strongly affected abundance of herbivores feeding on the flowers than those feeding on leaves. Infestation with caterpillars followed by aphids caused an increase in flower-feeding herbivores compared to the reversed order of infestation in one plant population, whereas the opposite effects were observed for the other plant population. The impact on plant seed set in the first reproductive year was limited. Our work shows that the identity and arrival order of early season herbivores may have long-term consequences for community composition on individual plants and that these patterns may vary among plant populations. We discuss how these community processes may affect plant fitness and speculate on the implications for evolution of plant defences.