To examine the independent association between physical activity and subclinical cerebrovascular disease as measured by silent brain infarcts (SBI) and white matter hyperintensity volume (WMHV). The Northern Manhattan Study (NOMAS) is a population-based prospective cohort examining risk factors for incident vascular disease, and a subsample underwent brain MRI. Our primary outcomes were SBI and WMHV. Baseline measures of leisure-time physical activity were collected in person. Physical activity was categorized by quartiles of the metabolic equivalent (MET) score. We used logistic regression models to examine the associations between physical activity and SBI, and linear regression to examine the association with WMHV. There were 1,238 clinically stroke-free participants (mean age 70 ± 9 years) of whom 60% were women, 65% were Hispanic, and 43% reported no physical activity. A total of 197 (16%) participants had SBI. In fully adjusted models, compared to those who did not engage in physical activity, those in the upper quartile of MET scores were almost half as likely to have SBI (adjusted odds ratio 0.6, 95% confidence interval 0.4-0.9). Physical activity was not associated with WMHV. Increased levels of physical activity were associated with a lower risk of SBI but not WMHV. Engaging in moderate to heavy physical activities may be an important component of prevention strategies aimed at reducing subclinical brain infarcts.