The pollution of agricultural soils by the heavy metals affects the productivity of the land and has an impact on the quality of the surrounding ecosystems. The present study investigated the bacterial community structure in the heavy metal contaminated sites along a smelter and a distantly located paddy field to elucidate the factors that are related to the alterations of the bacterial communities under the conditions of heavy metal pollution. Among the study sites, the bacterial communities in the soil did not show any significant differences in their richness and diversity. The soil bacterial communities at the three study sites were distinct from one another at each site, possessing a distinct set of bacterial phylotypes. Among the study sites, significant changes were observed in the abundances of the bacterial phyla and genera. The variations in the bacterial community structure were mostly related to the general soil properties at the phylum level, while at the finer taxonomic levels, the concentrations of arsenic (As) and lead (Pb) were the significant factors, affecting the community structure. The relative abundances of the genera Desulfatibacillum and Desulfovirga were negatively correlated to the concentrations of As, Pb, and cadmium (Cd) in the soil, while the genus Bacillus was positively correlated to the concentrations of As and Cd. According to the results of the prediction of bacterial community functions, the soil bacterial communities of the heavy metal polluted sites were characterized by the more abundant enzymes, involved in DNA replication and repair, translation, transcription, and the nucleotide metabolism pathways, while the amino acid and lipid metabolism, as well as the biodegradation potential of xenobiotics, were reduced. Our results showed that the adaptation of the bacterial communities to the heavy metal contamination was predominantly attributed to the replacement process, while the changes in community richness were linked to the variations in the soil pH values.