This paper reports the synthesis of alkanethiolate-functionalized core/shell Fe3O4/Au nanoparticles (Fe3O4@Au NPs) that combine the advantages of core/shell magnetic nanoparticles with self-assembled monolayers (SAMs). The alkanethiolates, having carboxylic acid (COOH) and methyl (CH3) terminal groups, can be easily self-assembled on the Fe3O4@Au NPs substrates. The surface composition and structure of 11-mercaptoundecanoic acid (11-MUA) and 1-dodecanethiol (11-DDT) SAMs were characterized in detail using energy-dispersive spectroscopy (EDS), Fourier transform infrared spectroscopy (FTIR), and X-ray photoelectron spectroscopy (XPS). To further study the surface properties of the SAMs, hydrophobic, ionic, and nonionic organic compounds were selected as probe molecules. The results indicated that the hydrophobic interaction caused strong retention of analytes, whereas the repulsive electrostatic interaction between the negatively charged 11-MUA SAMs surface and the negatively charged probe molecule would lead to the decrease of adsorption affinity.