Primary age-related tauopathy (PART) is a form of Alzheimer-type neurofibrillary degeneration occurring in the absence of amyloid-beta (Aβ) plaques. While PART shares some features with Alzheimer disease (AD), such as progressive accumulation of neurofibrillary tangle pathology in the medial temporal lobe and other brain regions, it does not progress extensively to neocortical regions. Given this restricted pathoanatomical pattern and variable symptomatology, there is a need to reexamine and improve upon how PART is neuropathologically assessed and staged. We performed a retrospective autopsy study in a collection (n = 174) of post-mortem PART brains and used logistic regression to determine the extent to which a set of clinical and neuropathological features predict cognitive impairment. We compared Braak staging, which focuses on hierarchical neuroanatomical progression of AD tau and Aβ pathology, with quantitative assessments of neurofibrillary burden using computer-derived positive pixel counts on digitized whole slide images of sections stained immunohistochemically with antibodies targeting abnormal hyperphosphorylated tau (p-tau) in the entorhinal region and hippocampus. We also assessed other factors affecting cognition, including aging-related tau astrogliopathy (ARTAG) and atrophy. We found no association between Braak stage and cognitive impairment when controlling for age (p = 0.76). In contrast, p-tau burden was significantly correlated with cognitive impairment even when adjusting for age (p = 0.03). The strongest correlate of cognitive impairment was cerebrovascular disease, a well-known risk factor (p < 0.0001), but other features including ARTAG (p = 0.03) and hippocampal atrophy (p = 0.04) were also associated. In contrast, sex, APOE, psychiatric illness, education, argyrophilic grains, and incidental Lewy bodies were not. These findings support the hypothesis that comorbid pathologies contribute to cognitive impairment in subjects with PART. Quantitative approaches beyond Braak staging are critical for advancing our understanding of the extent to which age-related tauopathy changes impact cognitive function.