The effective population size (Ne) is a central factor in determining maintenance of genetic variation. The neutral theory predicts that loss of variation depends on Ne, with less genetic drift in larger populations. We monitored genetic drift in 42 Drosophila melanogaster populations of different adult census population sizes (10, 50 or 500) using pooled RAD sequencing. In small populations, variation was lost at a substantially lower rate than expected. This observation was consistent across two ecological relevant thermal regimes, one stable and one with a stressful increase in temperature across generations. Estimated ratios between Ne and adult census size were consistently higher in small than in larger populations. The finding provides evidence for a slower than expected loss of genetic diversity and consequently a higher than expected long-term evolutionary potential in small fragmented populations. More genetic diversity was retained in areas of low recombination, suggesting that associative overdominance, driven by disfavored homozygosity of recessive deleterious alleles, is responsible for the maintenance of genetic diversity in smaller populations. Consistent with this hypothesis, the X-chromosome, which is largely free of recessive deleterious alleles due to hemizygosity in males, fits neutral expectations even in small populations. Our experiments provide experimental answers to a range of unexpected patterns in natural populations, ranging from variable diversity on X-chromosomes and autosomes to surprisingly high levels of nucleotide diversity in small populations.