Fossils are the only remaining evidence of the majority of species that have ever existed, providing a direct window into events in evolutionary history that shaped the diversification of life on Earth. Phylogenies underpin our ability to make sense of evolution but are routinely inferred using only data available from living organisms. Although extinct taxa have been shown to add crucial information for inferring macroevolutionary patterns and processes (such as ancestral states, paleobiogeography and diversification dynamics), the role fossils play in reconstructing phylogeny is controversial. Since the early years of phylogenetic systematics, different studies have dismissed the impact of fossils due to their incompleteness, championed their ability to overturn phylogenetic hypotheses or concluded that their behavior is indistinguishable from that of extant taxa. Based on taxon addition experiments on empirical data matrices, we show that the inclusion of paleontological data has a remarkable effect in phylogenetic inference. Fossils often have higher levels of topological influence than extant taxa, while inducing unique topological rearrangements. Previous studies have proposed a suite of explanations for the topological behavior of fossils, such as their retention of unique morphologies or their ability to break long branches. We develop predictive models that demonstrate that the possession of distinctive character state combinations is the primary predictor of the degree of induced topological change, and that the relative impact of taxa (fossil and extant) can be predicted to some extent before any phylogenetic analysis. Our results bolster the consensus of recent empirical studies by showing the unique role of paleontological data in phylogenetic inference, and provide the first quantitative assessment of its determinants, with broad consequences for the design of taxon sampling in both morphological and total-evidence analyses.