Despite of a growing interest in considering the role of sociological factors in seed exchanges and their consequences on the evolutionary dynamics of agro-biodiversity, very few studies assessed the link between ethno-linguistic diversity and genetic diversity patterns in small-holder farming systems. This is key for optimal improvement and conservation of crop genetic resources. Here, we investigated genetic diversity at 17 SSR markers of pearl millet landraces (varieties named by farmers) in the Lake Chad Basin. 69 pearl millet populations, representing 27 landraces collected in eight ethno-linguistic farmer groups, were analyzed. We found that the farmers’ local taxonomy was not a good proxy for population’s genetic differentiation as previously shown at smaller scales. Our results show the existence of a genetic structure of pearl millet mainly associated with ethno-linguistic diversity in the western side of the lake Chad. It suggests there is a limit to gene flow between landraces grown by different ethno-linguistic groups. This result was rather unexpected, because of the highly outcrossing mating system of pearl millet, the high density of pearl millet fields all along the green belt of this Sahelian area and the fact that seed exchanges among ethno-linguistic groups are known to occur. In the eastern side of the Lake, the pattern of genetic diversity suggests a larger efficient circulation of pearl millet genes between ethno-linguistic groups that are less numerous, spatially intermixed and, for some of them, more prone to exogamy. Finally, other historical and environmental factors which may contribute to the observed diversity patterns are discussed.