Human brains are markedly asymmetric in structure and lateralized in function, which suggests a relationship between these two properties. The brains of other closely related primates, such as chimpanzees, show similar patterns of asymmetry, but to a lesser degree, indicating an increase in anatomical and functional asymmetry during hominin evolution. We analysed the heritability of cerebral asymmetry in chimpanzees and humans using classic morphometrics, geometric morphometrics, and quantitative genetic techniques. In our analyses, we separated directional asymmetry and fluctuating asymmetry (FA), which is indicative of environmental influences during development. We show that directional patterns of asymmetry, those that are consistently present in most individuals in a population, do not have significant heritability when measured through simple linear metrics, but they have marginally significant heritability in humans when assessed through three-dimensional configurations of landmarks that reflect variation in the size, position, and orientation of different cortical regions with respect to each other. Furthermore, genetic correlations between left and right hemispheres are substantially lower in humans than in chimpanzees, which points to a relatively stronger environmental influence on left–right differences in humans. We also show that the level of FA has significant heritability in both species in some regions of the cerebral cortex. This suggests that brain responsiveness to environmental influences, which may reflect neural plasticity, has genetic bases in both species. These results have implications for the evolvability of brain asymmetry and plasticity among humans and our close relatives.