The cortical motor system follows a modular organization in which different features of executed movements are supported by distinct streams. Accordingly, different levels of action recognition, such as movement characteristics or action semantics may be processed within distinct networks. The present study aimed to differentiate areas related to the analysis of action features involving semantic knowledge from regions concerned with the evaluation of movement characteristics determined by structural object properties. To this end, the assessment of (i) tool-associated actions in relation to semantically, but not functionally inappropriate recipients (factor "Semantics"), and the evaluation of (ii) tool-associated movements performed with awkward versus correct hand postures (factor "Hand") were experimentally manipulated in an fMRI study with an event-related 2 × 2 factorial design. The videos used as stimuli displayed actions performed with the right hand in third-person perspective. Conjunction analysis of all four experimental conditions showed that observing videos depicting tool-related actions compared to rest was associated with widespread bilateral activity within the frontal lobes, inferior and superior parietal lobules, parts of the temporal lobes, as well as the occipital lobes. Viewing actions executed with incorrect compared to correct hand postures (factor "Hand") elicited significantly more activity within right primary sensory cortex (Brodmann area 2) and superior parietal lobule. Conversely, tool-associated actions displayed after semantically incorrect compared to correct recipients elicited higher activation within a left-lateralized network comprising the ventro-lateral prefrontal cortex (VLPFC), parts of the intraparietal sulcus and the angular gyrus (AG), as well as the supplementary motor area (SMA) and pre-SMA. Probabilistic diffusion tensor imaging-based tractography revealed two distinct fiber connections between AG and the frontal activation: A dorsal pathway via the superior longitudinal fascicle to the caudal part of VLPFC and a ventral pathway reaching the more rostral parts of VLPFC via the extreme capsule. The task-dependent relative modulation of activity within these brain networks composed of activated cortical areas connected by specific white matter tracts may indicate that the assessment of semantic action features relies on both dorso-ventral and ventral processing streams, whereas the analysis of hand postures with respect to objects depends on areas within the dorso-dorsal stream.