The rarity of nitrogen (N)-fixing trees in frequently N-limited higher-latitude (here, > 35°) forests is a central biogeochemical paradox. One hypothesis for their rarity is that evolutionary constraints limit N-fixing tree diversity, preventing N-fixing species from filling available niches in higher-latitude forests. Here, we test this hypothesis using data from the USA and Mexico. N-fixing trees comprise only a slightly smaller fraction of taxa at higher vs. lower latitudes (8% vs. 11% of genera), despite 11-fold lower abundance (1.2% vs. 12.7% of basal area). Furthermore, N-fixing trees are abundant but belong to few species on tropical islands, suggesting that low absolute diversity does not limit their abundance. Rhizobial taxa dominate N-fixing tree richness at lower latitudes, whereas actinorhizal species do at higher latitudes. Our results suggest that low diversity does not explain N-fixing trees' rarity in higher-latitude forests. Therefore, N limitation in higher-latitude forests likely results from ecological constraints on N fixation.