1. Darwin's naturalisation hypothesis posing that phylogenetic distance of alien species to native residents predicts invasion success, and Elton's diversity‐invasibility hypothesis posing that diversity of native communities confers resistance to invasion, are both rooted in ideas of species coexistence. Because the two hypotheses are inherently linked, the mechanisms underlying them may interact in driving the invasion success. Even so, these links and interactions have not been explicitly disentangled in one experimental study before. 2. To disentangle the links between the two hypotheses, we used 36 native grassland herbs to create greenhouse mesocosms with 90 grassland communities of different diversities, and introduced each of five herbaceous alien species as seeds and seedlings. We used phylogeny and four functional traits (plant height, specific leaf area, leaf size, seed mass) to calculate different measures of phylogenetic and functional distance and diversity. Specifically, we tested how the alien‐native distance (phylogenetic or functional) and the native diversity (phylogenetic or functional) affected each other in their effects on germination, seedling survival, growth and reproduction of the aliens. 3. Overall, our results supported both hypotheses. Multivariate functional distance based on four traits jointly had stronger positive effects than phylogenetic distance and the univariate ones based on each trait separately. Moreover, the aliens were more successful if they were more competitive by being taller and having larger leaves with a lower SLA than the native residents. Univariate functional diversity based on each trait separately had stronger negative effects than phylogenetic and multivariate functional diversity. Most importantly, we found that the effects of alien‐native phylogenetic and multivariate functional distance became stronger as diversity increased. Our analyses with single traits also showed that the strength of the effects of both alien‐native hierarchical functional distances (indicative of competitive inequalities) and absolute functional distances (indicative of niche differences) increased at higher diversities, where competition is more severe. 4. SynthesisOur study explicitly demonstrates for the first time how the mechanisms underlying the two classical invasion hypotheses interact in driving invasion success in grassland communities. This may help to explain some of the puzzling results of studies testing either of the two hypotheses.