Human cultural development is marked by the emergence of new words and ideas, reflecting societal changes. But how does this evolution proceed? We use modern methods in natural language processing (namely, word embeddings) to measure statistical traces of cultural development, providing a testing ground to compare different models as to how this process works. We show that real embeddings of English and 21 other languages exhibit a series of previously unrecognized regularities, specifically (a) frequency assortativity, where entities of high popularity cluster near other high-popularity entities, (b) characteristic clustering velocity profiles due to aggregation into hierarchical structures, (c) persistent temporal dynamics, where newly-created entities appear disproportionately near other recent entries, and (d) Taylor’s law, implying that over time and across empirical semantic space the variance in new entity counts scales as a power of the mean, which helps systematize and quantify large historical fluctuations of neologisms. To explain these facts, we propose a class of generative models (specifically, directed preferential placement) that construct synthetic embeddings exhibiting similar regularities. We show that analogous regularities also occur in other data sets, suggesting that such generating models may shed light on new aspects of language and cultural evolution.