The development of organisms and tissues is dictated by an elaborate balance between cell division, apoptosis and differentiation: the cell population dynamics. To quantify these dynamics, we propose a phylodynamic inference approach based on data from single-cell lineage recorders. For this purpose, we developed a Bayesian phylogenetic framework, TiDeTree, that uses lineage recorder data as input to estimate time-scaled single-cell trees. By implementing TiDeTree in the BEAST 2 platform, we enable joint inference of the time-scaled trees and, for the first time, the cell population dynamics. We validated TiDeTree using simulations and showed that its performance can further be improved by including multiple sources of additional independent information into the inference, such as frequencies of editing outcomes or experimental replicates. We benchmarked TiDeTree against state-of-the-art methods and show that it performs comparably in terms of tree topology estimation and additionally enables direct assessment of uncertainty, estimation of a time-scaled tree, and co-estimation of additional parameters. To demonstrate TiDeTree’s use in practice, we analysed a public data set containing lineage recorder data from ~100 stem cell colonies. We estimated a time-scaled phylogeny for each colony, as well as the cell division and apoptosis rates underlying the growth dynamics of all colonies. We envision that TiDeTree will find broad application in the analysis of single-cell lineage tracing data, which will improve our understanding of cellular processes during development.