The association between serum free hemoglobin (sfHb) level and white matter hyperintensity (WMH) volume is controversial. This study is to examine this association considering nonlinearity, sex dimorphism, and WMH type. We enrolled 704 older adults among the participants of the Korean Longitudinal Study on Cognitive Aging and Dementia and visitors to the Dementia Clinic of Seoul National University Bundang Hospital. We measured sfHb level in the venous blood and WMH volume (VWMH) using fluid-attenuated inversion recovery magnetic resonance images. The association between sfHb level and periventricular VWMH was linear in men (linear regression; β = − 0.18, p = 0.006) and U-shaped in women (restricted cubic spline; F = 6.82, p < 0.001). sfHb level was not associated with deep VWMH in either sex. These findings were also observed in participants without anemia. To conclude, sfHb level is associated with periventricular VWMH in older adults of both sexes. Maintaining an optimal sfHb level may contribute to the prevention of WMH.