Objective: To assess the association of early morning serum cortisol with cognitive performance and brain structural integrity in community-dwelling young and middle-aged adults without dementia. Methods: We evaluated dementia-free Framingham Study (Generation 3) participants (mean age 48.5 years; 46.8% men), who underwent cognitive testing for memory, abstract reasoning, visual perception, attention, and executive function (n= 2231), and brain MRI (n=2018) to assess total white matter, lobar gray matter, and white matter hyperintensity volumes and fractional anisotropy (FA) measures. We used linear and logistic regression to assess the relations of cortisol (categorized in tertiles, with the middle tertile as referent) to measures of cognition, MRI volumes, presence of covert brain infarcts (CBI) and cerebral microbleeds (CMB), and voxel-based microstructural white matter integrity and gray matter density, adjusting for age, sex, APOE and vascular risk factors. Results: Higher cortisol (highest tertile vs. middle tertile) was associated with worse memory and visual perception, as well as lower total cerebral brain, occipital and frontal lobar gray matter volumes. Higher cortisol was associated with multiple areas of microstructural changes (decreased regional FA), especially in the splenium of corpus callosum and the posterior corona radiata. The association of cortisol with total cerebral brain volume varied by sex (p interaction=0.048); higher cortisol was inversely associated with cerebral brain volume in women [p=0.001] but not in men [p=0.717]). There was no effect modification by the apoE4 genotype of the relations of cortisol and cognition or imaging traits. Conclusions: Higher serum cortisol was associated with lower brain volumes and impaired memory in asymptomatic younger to middle-aged adults, with the association being evident particularly in women.