The optical manipulation of nanoscale objects via structured light has attracted significant attention for its various applications, as well as for its fundamental physics. In such cases, the detailed behavior of nano-objects driven by optical forces must be precisely predicted and controlled, despite the thermal fluctuation of small particles in liquids. In this study, the optical forces of an optical vortex acting on gold nanoparticles (Au NPs) are visualized using dark-field microscopic observations in a nanofluidic channel with strictly suppressed forced convection. Manipulating Au NPs with an optical vortex allows the evaluation of the three optical force components, namely, gradient, scattering, and absorption forces, from the in-plane trajectory. We develop a Langevin dynamics simulation model coupled with Rayleigh scattering theory and compare the theoretical results with the experimental ones. Experimental results using Au NPs with diameters of 80–150 nm indicate that our experimental method can determine the radial trapping stiffness and tangential force with accuracies on the order of 0.1 fN/nm and 1 fN, respectively. Our experimental method will contribute to broadening not only applications of the optical-vortex manipulation of nano-objects, but also investigations of optical properties on unknown nanoscale materials via optical force analyses.