The Tropical Andes are Earth's most species-rich biodiversity hotspot for both animals and plants. Nevertheless, the ecological and evolutionary processes underlying this extraordinary richness remain uncertain. Here, we examine the processes that generate high richness in the Andes relative to other regions in South America, and across different elevations within the Andes, using frogs as a model system. We combine distributional data, a newly generated time-calibrated phylogeny for 2318 frog species, and phylogenetic comparative methods to test the relative importance of diversification rates and colonization times for explaining Andean diversity at different scales. At larger scales (among regions and families), we find that faster diversification rates in Andean clades most likely explain high Andean richness. In contrast, at smaller temporal and spatial scales (within family-level clades, within the Andes), diversification rates rarely explain richness patterns. Instead, we show that colonization times are important for shaping elevational richness patterns within the Andes, with more species found in habitats colonized earlier. We suggest that these scale-dependent patterns might apply to many other richness gradients. Recognition of this scale-dependence may help to reconcile conflicting results among studies of richness patterns across habitats, regions, and organisms.