We study the optical material contrast of single nanoparticles in infrared scattering-type near-field optical microscopy (IR s-SNOM) in the presence of strong probe-substrate coupling. It is shown theoretically and experimentally that the contrast depends on both the dielectric properties of the nanoparticles and on their size. We can separate the two dependencies by correlating the simultaneously acquired topography and near-field images pixel-by-pixel. This allows us to establish material-specific mapping of polydisperse nanoparticle mixtures with nanoscale spatial resolution. We experimentally demonstrate the differentiation between sub-10 nm gold and polymer particles adsorbed on a Si substrate. Possible applications of our method range from the material-specific mapping of nanoparticle assemblies to the measurement of the doping concentration in single semiconductor nanoparticles.