Relaxed clock methods account for among-branch-rate-variation when estimating divergence times by inferring different rates for individual branches. In order to infer different rates for individual branches, important assumptions are required. This is because molecular sequence data does not provide direct information about rates, but instead provides direct information about the total number of substitutions along any branch, which is a product of the rate and time for that branch. Often, the assumptions required for estimating rates for individual branches depend heavily on the implementation of multiple fossil calibrations in a single phylogeny. Here, we show that the basis of these assumptions is often critically undermined. First, we highlight that the temporal distribution of the fossil record often violates key assumptions of methods that use multiple fossil calibrations with relaxed clocks. With respect to “node calibration” methods, this conclusion is based on our inference that different fossil calibrations are unlikely to reflect the relative ages of different clades. With respect to the fossilised-birth-death-process, this conclusion is based on our inference that the fossil recovery rate is often highly heterogeneous. We then demonstrate that methods of divergence time estimation that use multiple fossil calibrations are highly sensitive to assumptions about the fossil record and among-branch-rate-variation. Given the problems associated with these assumptions, our results highlight that using multiple fossil calibrations with relaxed clocks often does little to improve the accuracy of divergence time estimates.