It is well-established that brain structures and cognitive functions change across the lifespan. A longstanding hypothesis called age differentiation additionally posits that the relations between cognitive functions also change with age. To date however, evidence for age-related differentiation is mixed, and no study has examined differentiation of the relationship between brain and cognition. Here we use multi-group Structural Equation Modeling and SEM Trees to study differences within and between brain and cognition across the adult lifespan (18-88 years) in a large (N>646, closely matched across sexes), population-derived sample of healthy human adults from the Cambridge Centre for Ageing and Neuroscience (www.camcan. org). After factor analyses of grey-matter volume (from T1- and T2-weighted MRI) and white-matter organisation (fractional anisotropy from Diffusion-weighted MRI), we found evidence for differentiation of grey and white matter, such that the covariance between brain factors decreased with age. However, we found no evidence for age differentiation between fluid intelligence, language and memory, suggesting a relatively stable covariance pattern between cognitive factors. Finally, we observed a specific pattern of age differentiation between brain and cognitive factors, such that a white matter factor, which loaded most strongly on the hippocampal cingulum, became less correlated with memory performance in later life. These patterns are compatible with reorganization of cognitive functions in the face of neural decline, and/or with the emergence of specific subpopulations in old age.