The sex chromosomes have been hypothesized to play a key role in driving adaptation and speciation across many taxa. The reason for this is thought to be the hemizygosity of the heteromorphic part of sex chromosomes in the heterogametic sex, which exposes recessive mutations to natural and sexual selection. The exposure of recessive beneficial mutations increases their rate of fixation on the sex chromosomes, which results in a faster rate of evolution. In addition, genetic incompatibilities between sex-linked loci are exposed faster in the genomic background of hybrids of divergent lineages, which makes sex chromosomes contribute disproportionately to reproductive isolation. However, in birds, which show a Z/W sex determination system, the role of adaptation vs. genetic drift as the driving force of the faster differentiation of the Z chromosome (fast-Z effect) and the disproportionate role of the Z chromosome in reproductive isolation (large-Z effect) are still debated. Here, we address this debate in the bird genus Ficedula flycatchers based on population-level whole-genome sequencing data of six species. Our analysis provides evidence for both faster lineage sorting and reduced gene flow on the Z chromosome than the autosomes. However, these patterns appear to be driven primarily by the increased role of genetic drift on the Z chromosome, rather than an increased rate of adaptive evolution. Genomic scans of selective sweeps and fixed differences in fact suggest a reduced action of positive selection on the Z-chromosome. Nevertheless, it is possible that the faster lineage sorting of the Z chromosome due to genetic drift may help drive the evolution of genetic incompatibilities between species.