Variance reduction stochastic gradient descent methods enable minimization of model fitting problems involving big datasets with low iteration complexity and fast asymptotic convergence rates. However, they scale poorly in distributed settings. In this paper, we propose a highly parallel variance reduction method, CentralVR, with performance that scales linearly with the number of worker nodes. We also propose distributed versions of popular variance reduction methods that support a high degree of parallelization. Unlike existing distributed stochastic gradient schemes, CentralVR exhibits linear performance gains up to thousands of cores for massive datasets.