Watching a rubber hand being stroked by a paintbrush while feeling identical stroking of one's own occluded hand can create a compelling illusion that the seen hand becomes part of one's own body. It has been suggested that this so-called rubber hand illusion (RHI) does not simply reflect a bottom-up multisensory integration process but that the illusion is also modulated by top-down, cognitive factors. Here we investigated for the first time whether the conceptual interpretation of the sensory quality of the visuotactile stimulation in terms of roughness can influence the occurrence of the illusion and vice versa, whether the presence of the RHI can modulate the perceived sensory quality of a given tactile stimulus (i.e., in terms of roughness). We used a classical RHI paradigm in which participants watched a rubber hand being stroked by either a piece of soft or rough fabric while they received synchronous or asynchronous tactile stimulation that was either congruent or incongruent with respect to the sensory quality of the material touching the rubber hand. (In)congruencies between the visual and tactile stimulation did neither affect the RHI on an implicit level nor on an explicit level, and the experience of the RHI in turn did not cause any modulations of the felt sensory quality of touch on participant's own hand. These findings first suggest that the RHI seems to be resistant to top-down knowledge in terms of a conceptual interpretation of tactile sensations. Second, they argue against the hypothesis that participants own hand tends to disappear during the illusion and that the rubber hand actively replaces it.