Both natural and anthropogenic factors can negatively affect species demography and even lead to species extinction, but their relative importance in driving species rarity remains poorly understood. To this end, we investigated the phylogeography and demographic history of the endemic endangered conifer species Torreya jackii. We collected samples from all the 13 extant wild populations, and analyzed the genetic variation using 8 nuclear microsatellites and 4 chloroplast and 1 mitochondrial DNA fragments. Low genetic and nucleotide diversity was revealed, which could explain the absence of spatial and phylogeographic structure. Using a hierarchical approximate Bayesian computation technique, we identified the demographic scenario that best fit with the genetic data, where the effective population size of the plant was low at least 200,000 years ago but expanded after the last glacial maximum (LGM). The scenarios accounting for population decline during the LGM had significantly smaller posterior probabilities, suggesting in situ survival of this species during the LGM. The palaeoclimatic niche model revealing a profound impact of precipitation on the distribution of T. jackii also predicted that the current distribution areas were suitable during the LGM. Although the post-LGM expansion pulled T. jackii from the brink of long-term rarity, the best-supported scenario nevertheless indicated population collapse during recent centuries with a population size reduction by more than 80%, bringing the plant back to the brink. Our study thus sheds light on how historical factors and human impacts jointly threaten the persistence of species and should be considered in biodiversity conservation.