Dogs are among the most variable species today, but little is known about the morphological variability in the early phases of their history. The Neolithic transition to farming may have resulted in an early morphological diversification as a result of changes in the anthropic environment or intentional selection on specific morphologies. Here, we explore the variability and modularity in mandible form by comparing 525 dog mandibles from European archaeological sites ranging from 8,100 to 3,000 cal. BC to a reference sample of modern dogs, wolves, and dingoes. We use three-dimensional geometric morphometrics to quantify the form of complete and fragmented mandibles. We demonstrate that an important morphological variability already existed before the Bronze Age in Europe, yet the largest, smallest, most brachycephalic or dolichocephalic extant dogs have no equivalent in the archaeological sample, resulting in a lower variation compared to modern relatives. The covariation between the anterior and posterior parts of the mandible is lower in archaeological dogs, suggesting a low degree of intentional human selection in early periods. The mandible of modern and ancient dogs differs in functionally important areas, possibly reflecting differences in diet, competition, or the implication of ancient dogs in hunting or defense.