Current biodiversity loss was generally considered to be caused by anthropogenic disturbance, but when anthropogenic activities began to impact biodiversity loss is still controversial. One hypothesis suggested it was from the industrial era, while others proposed that the anthropogenic disturbance had already resulted in biodiversity decline since the Early Holocene. To test these hypotheses, we focus on subtropical East Asia, which has witnessed a land use and anthropogenic history since the Early Holocene. We selected the unique vegetation of evergreen broadleaved forests (EBLFs), using a genomic approach to infer the demographic history of a dominant plant (Litsea elongata) of EBLFs, and to further detangle the impact of climate change and/or anthropogenic disturbance on effective population size fluctuation. Nine well-defined geographical clades were identified within extant populations of L. elongata. The estimated historical population sizes of these clades all contracted, indicating persistent population decline over the last 11,000 years. Significant correlation was detected between demographic history and three anthropogenic disturbance factors, rather than the climate change in the Holocene. Therefore, we provided the dataset used for phylogenomic analysis,  demographic history inference, and correlation analysis in this study.