Objective: To evaluate the association of cognitive declines in the domains of memory, language, and executive function with brain gray matter (GM) volume in old age. Methods: Prospective study of 1,846 participants in the Atherosclerosis Risk in Communities (ARIC) study who underwent 3T brain MRI scans in 2011-2013. Participants were categorized by cognitive domain performance trajectory over the prior 20 years (cut-point to define decline: 20th percentile). Associations between GM volume and cognitive declines were assessed at the voxel level using voxel-based morphometry and at the regional level using atlas-defined GM volumes of specific regions of interest. Results: Participants were an average age of 76 years, 60% were female, and 28% were black. Participants in the top 20th percentile for decline in the memory domain had smaller GM volumes in the medial temporal lobe (-3.3%, 95% CI: -4.6% to -2.1%), amygdala (-2.7%, 95% CI: -4.1% to -1.3%), entorhinal cortex (-4.1%, 95% CI: -6.0% to -2.2%), and hippocampus (-3.8%, 95% CI: -5.2% to -2.4%) compared to participants who were in the lowest 80th percentile for decline in all domains. In contrast, among participants who were in the top 20th percentile for decline in the language or executive function domains, GM volumes were smaller in more brain regions. Conclusions: Declines in memory function were associated with brain volume loss in the medial temporal and hippocampal formations. Declines in language and executive function were associated with decreases in brain volumes across more non-contiguous brain regions.