APOE ε4 is a well-established risk factor for cognitive impairment and decline in old age. It is an open question, however, if such decline already commences in midlife and if APOE ε4 influences other factors, including flexibility and well-being. In this study, we apply growth curve models to compare longitudinal change trajectories from age 45 to 85 on fluid and crystallized abilities, flexibility, and well-being between APOE ε4 carriers (N = 230) and non-carriers (N = 591) from the Seattle Longitudinal Study. Our results reveal steeper decline for APOE ε4 carriers on reasoning, numeric ability, and verbal meaning. No evidence was found for associations of APOE to flexibility and well-being or that these factors moderate cognitive decline among APOE ε4 carriers. We take these findings to suggest that decline in specific cognitive abilities may occur earlier than typically thought and discuss routes for further inquiry, such as possible intervention implications.