Reasons why chromosomal rearrangements spread to fixation and frequently distinguish related taxa remain poorly understood. We used cytological descriptions of karyotype to identify large pericentric inversions between species of Estrildid finches (family Estrildidae) and a time-dated phylogeny to assess the genomic, geographic, and phylogenetic context of karyotype evolution in this group. Inversions between finch species fixed at an average rate of one every 2.26 My. Inversions were twice as likely to fix on the sex chromosomes compared to the autosomes. A high repeat density on the sex chromosomes may increase mutation rates, but other explanations via mutagenic input are not supported, as the number of inversions on a chromosome does not correlate with its length or map size. Inversions have fixed 3.3× faster in three continental clades than in two island chain clades, and fixation rate correlates with both range size and the number of sympatric species pairs. These results point to adaptation as the dominant mechanism driving fixation and suggest a role for gene flow in karyotype divergence. A review shows that the rapid karyotype evolution observed in the Estrildid finches appears to be more general across birds, and by implication other understudied taxa.