The inferior parietal lobule (IPL) is one of the most expanded cortical regions in humans relative to other primates. It is also among the most structurally and functionally asymmetric regions in the human cerebral cortex. Whether the structural and connectional asymmetries of IPL subdivisions differ across primate species and how this relates to functional asymmetries remain unclear. We identified IPL subregions that exhibited positive allometric in both hemispheres, scaling across rhesus macaque monkeys, chimpanzees, and humans. The patterns of IPL subregions asymmetry were similar in chimpanzees and humans, but no IPL asymmetries were evident in macaques. Among the comparative sample of primates, humans showed the most widespread asymmetric connections in the frontal, parietal, and temporal cortices, constituting leftward asymmetric networks that may provide an anatomical basis for language and tool use. Unique human asymmetric connectivity between the IPL and primary motor cortex might be related to handedness. These findings suggest that structural and connectional asymmetries may underlie hemispheric specialization of the human brain.