Memory problems are among the most common complaints as people grow older. Using structural equation modeling of commensurate scores of anterograde memory from a large (N = 315), population-derived sample (www.cam-can.org), we provide evidence for three memory factors that are supported by distinct brain regions and show differential sensitivity to age. Associative memory and item memory are dramatically affected by age, even after adjusting for education level and fluid intelligence, whereas visual priming is not. Associative memory and item memory are differentially affected by emotional valence, and the age-related decline in associative memory is faster for negative than for positive or neutral stimuli. Gray-matter volume in the hippocampus, parahippocampus and fusiform cortex, and a white-matter index for the fornix, uncinate fasciculus and inferior longitudinal fasciculus, show differential contributions to the three memory factors. Together, these data demonstrate the extent to which differential ageing of the brain leads to differential patterns of memory loss.The Cambridge Centre for Ageing and Neuroscience (Cam-CAN) research was supported by the Biotechnology and Biological Sciences Research Council (BB/H008217/1); R.N.H., S.E. and T.E. are additionally supported by the UK Medical Research Council (MC_A060_5PR10). RAK is supported by the Wellcome Trust (grant number 107392/Z/15/Z and the UK Medical Research Council (MC-A060-5PR61). We are grateful to the Cam-CAN respondents and their primary care teams in Cambridge for their participation in this study. We also thank colleagues at the MRC Cognition and Brain Sciences Unit MEG and MRI facilities for their assistance.