The CHA2DS2-VASc score is a validated predictor of ischemic stroke in atrial fibrillation (AF) patients. However, data are limited on whether the CHA2DS2-VASc score is associated with subclinical brain structural changes or physical frailty among older AF patients. We assessed the relationship between CHA2DS2-VASc scores and brain structural changes or physical frailty in AF patients without history of stroke. Overall, 117 patients completed a comprehensive geriatric assessment for physical frailty. In brain magnetic resonance imaging sub-study (n = 49), brain volume and white matter hyperintensity lesion burden were automatically quantified using the LESIONQUANT software program. Patients with high risk of CHA2DS2-VASc scores (≥ 2 in men or ≥ 3 in women) tended to be older and had more comorbidities, higher frailty index, and slower gait speed. Total white matter hyperintensity lesion burden was higher in those with high risk of CHA2DS2-VASc score than in those with intermediate risk (score of 1 in men or 2 in women) of CHA2DS2-VASc score (1.67 [interquartile range: 0.70–3.45] vs. 0.64 [0.19–1.44], p = 0.036). Cognitive function was associated with brain volume, but gait speed was related with white matter hyperintensity lesion burden. In conclusion, we showed a positive relationship between CHA2DS2-VASc scores, white matter hyperintensity lesion burden, and physical frailty in older AF patients. Subclinical brain changes associated with high CHA2DS2-VASc scores may predict physical frailty risk.