Rapid diversification is often observed when founding species invade isolated or newly formed habitats that provide ecological opportunity for adaptive radiation. However, most of the Earth’s diversity arose in diverse environments where ecological opportunities appear to be more constrained. Here, we present a striking example of a rapid radiation in a highly diverse marine habitat. The hamlets, a group of reef fishes from the wider Caribbean, have radiated into a stunning diversity of color patterns but show low divergence across other ecological axes. Although the hamlet lineage is approximately 26 million years old, the radiation appears to have occurred within the last 10,000 generations in a burst of diversification that ranks among the fastest in fishes. As such, the hamlets provide a compelling backdrop to uncover the genomic elements associated with phenotypic diversification and an excellent opportunity to build a broader comparative framework for understanding the drivers of adaptive radiation. The analysis of 170 genomes suggests that color pattern diversity is generated by different combinations of alleles at a few large-effect loci. Such a modular genomic architecture of diversification has been documented before in Heliconius butterflies, capuchino finches and munia finches, three other tropical radiations that took place in highly diverse and complex environments. The hamlet radiation also occurred in a context of high effective population size, which is typical of marine populations. This allows for the accumulation of new variants through mutation and the retention of ancestral genetic variation, both of which appear to be important in this radiation.