Objective: To examine the association between systemic inflammation measured during midlife and 20-year cognitive decline. Methods: Within the Atherosclerosis Risk in Communities cohort study, inflammatory biomarkers were measured during middle adulthood. We created an inflammation composite score using four blood biomarkers measured at Visit 1 (fibrinogen, white blood cell count, von Willebrand factor, and Factor VIII); we measured C-reactive protein (CRP) at Visit 2. Cognition was assessed over three visits spanning 20 years using measures of memory, executive function, and language. Results: 12,336 participants (baseline age 56.8 [5.7], 21% black, 56% women) were included. After adjusting for demographic variables, vascular risk factors and comorbidities, each standard deviation (SD) increase in midlife inflammation composite score was associated with an additional 20-year decline of -0.035 SD (95% CI: -0.062, -0.007) on the cognitive composite score. We found a similar association between each SD increase in midlife CRP level and additional 20-year cognitive decline (-0.038 SD, 95% CI: -0.057, -0.019). Participants with a midlife inflammation composite score in the top quartile had a 7.8% steeper cognitive decline, compared to participants in the lowest quartile; CRP in the top quartile was associated with an 11.6% steeper cognitive decline. In cognitive domain-specific analyses, elevated midlife inflammatory markers were most consistently associated with declines in memory. Results were similar after adjusting for attrition using inverse probability weighting. Conclusions: Our findings highlight what may be an early pathogenic role for systemic inflammation as a driver of cognitive decline in the decades leading up to older adulthood.