Extant rear-edge populations located in former glacial refugia remain understudied despite their high conservation value. These populations should have experienced strong genetic drift due to their small size and long isolation. Moreover, the prolonged action of isolation-by-distance in refugial areas should result in stronger regional spatial genetic structure than in recolonized areas, but empirical tests of this prediction are scarce. To fill this gap, we first used a set of 16 microsatellite markers to investigate the genetic structure of European beech in France in 65 populations from three refugial areas and one control recolonized (non-refugial) area. Then, using the same approach, we reanalyzed published isozyme data from 375 populations distributed across the entire species range. We found stronger genetic differentiation among populations in refugia than in recolonized areas. However, contrary to expectations, regional spatial genetic structure was lower within refugia than within recolonized areas. Published studies presenting similar analyses suggest that our results could have generality across different biogeographical settings and types of organisms. Strong and prolonged genetic drift in refugial areas could have erased the signature of range expansions that is still visible in recolonized areas. Our results therefore suggest that Pleistocene population isolation has played a key role in increasing the genetic complexity of extant rear-edge populations.