The Digital Rubber Hand Illusion

The rubber hand illusion (RHI) is a body ownership illusion whereby congruently stroking a fake rubber hand and a subject’s hidden hand while observing the rubber hand produces the illusion of them feeling the touch on the rubber hand and experiencing the rubber hand to be part of their own body. The parameters of the RHI have not been fully defined and we describe an approach utilising digital media and technology to examine and establish parameters for this illusion beginning with three experiments: (i) Repeating the original RHI to determine if our test conditions are conducive to producing the classic illusion. (ii) Replacing the original rubber hand with either a static or animated digital image of it displayed on a tablet. (iii) A VR implementation whereby participants see an image of their own hand viewed through a head-mounted display. Measurements of proprioceptive drift (an objective indicator of the feeling of ownership of the rubber hand) corroborated the original RHI and also suggested that a similar phenomenon can occur when the rubber hand was replaced with either a static or animated digital image of it or in a VR condition.


INTRODUCTION
The rubber hand illusion (RHI) belongs to a category of illusions relating to body ownership where under specific multisensory conditions participants experience artificial body parts or bodies to be their own. The original demonstration of the RHI revealed that congruently stroking a subject's hidden hand and a fake rubber hand while the subject observed the latter produced the illusion of feeling the touch on the fake hand and experiencing the fake hand to be part of their own body. The illusion revealed a three-way interaction between vision, touch and proprioception and demonstrated that inter-modal matching is sufficient for self-attribution (Botvinick and Cohen, 1998).
The RHI has been reproduced on numerous occasions, often being conducted with variations in the nature of the visual or tactile stimuli used.
There are two competing explanations for this effect. In the first case, a review of body ownership illusion studies by Kilteni et al (2015) indicated that for illusions using visuotactile stimuli such as the RHI, spatiotemporal congruence between the seen and felt stimuli was sufficient to elicit the illusions in most experimental situations.
The second class of explanation relies on the plasticity of the "body schema" (e.g. Gallagher, 2005;Metzinger, 2004). This explanation of the RHI is based on the proposal that we maintain a representation (a schema) of the disposition of our limbs. This schema is plastic, changing as we grow (as children) and as we become less capable with age (the effects of arthritis) or accident (amputation). This plasticity comes into its own when we consider the example of the blind person walking with a white stick. When first presented with such a stick, it is experienced as an external tool projecting in front of us as we walk. However, with time, the stick "disappears" and the blind person experiences it as though it were part of their own body.
Irrespective of explanation, rubber hand illusion studies have demonstrated that visuomotor stimuli can induce the illusion of ownership of a fake body part by getting participants to perform active or passive movements in conjunction with the rubber hand (Kalckert and Ehrsson, 2012;2014).
As before, the successful operation of the illusion depended upon the spatiotemporal congruence of the stimulus (Singer & Gray, 1995). Consequently, a commonly used control treatment in these experiments has been asynchronous visuotactile or visuomotor stimulation which abolishes the illusion of body ownership (e.g. Botvinick and Cohen, 1998;Armel and Ramachandran, 2003;Ehrsson et al, 2004;Tsakiris and Haggard 2005;Slater et al, 2008). Spatial congruency of visuo-proprioceptive cues are not necessary if a fake body or limb is seen in an anatomically plausible configuration with congruent visuotactile or visuomotor stimulation. However, varying degrees of visuo-proprioceptive mismatch are able to modulate the onset and intensity of body ownership illusions.
The limitations of the RHI are not fully established, so our approach has been to utilise the potential of digital media as an experimental apparatus to identify and examine possible parameters relating to the efficacy of this illusion. Digital technologies such as virtual reality (VR) whereby an artificial or modulated environment created by computer software can be presented to a user in such a way that they make-believe it to be real offers the opportunity of creating an experimental system where stimulus variables can be manipulated independently, for example through the introduction of temporal delays. VR and mixed reality (MR) have been used to investigate the RHI. In a VR condition the rubber hand and its tactile stimulation were presented as a projection on the surface of the table in front of the participant. While in the MR condition the rubber hand was projected in front of the participant with the tactile stimulation applied directly to the projection. Both the VR and MR conditions were able to produce the illusion but not as strongly as the original (Ijsselstein et al, 2006). In the present work we describe a different implementation of VR where the participant's hand was captured and modified digitally, and presented to them through a head mounted display (HMD). VR with a HMD has been used successfully to demonstrate 'out of body illusions' where participants experience taking ownership of an external body including that of a toy doll and a giant (van der Hoort et al, 2011).

Experimental overview
Our digital approach to understanding and identifying parameters involved in the RHI can be described by three initial experiments. Participants in these were staff and students of a UK university who gave their informed consent to take part and no payments or rewards were made.

Experiment 1: Repeating the classical RHI
In order to evaluate responses to our digital interventions we needed a reference with which to compare our findings to and concluded that repeating the original RHI experiment described by Botvinick and Cohen (1998) was the most appropriate. Moreover, it was important to establish if our test conditions were conducive to creating the RHI before examining any digital interventions. Consequently, participants observed a rubber left hand being stroked and tapped by a paintbrush in synchrony with their own unseen left hand. The original internal control condition of asynchronous stroking/tapping between the participant's real hand and the rubber hand was used to determine any treatment effect. The experiment was performed with 32 participants (11 female, age range 18-34). Individual participants were seated at a table with their left arm resting on the table in a pronated position (palms down) and their forearm and index finger positioned over a marker. A standing screen was positioned beside the participant's left arm to hide it from their view and a life-size rubber model of a left hand and arm was placed over another marker in front of the participant (Figure 1). A measuring tape was attached to the side of the table facing the participant to record proprioceptive drift.
The participant was asked to fixate on the rubber hand while their hidden hand and the rubber hand were tapped or stroked with identical paintbrushes either synchronously (ca. 1 stroke/sec) in a spatially congruent manner or asynchronously (control condition). The experiment was conducted with independent samples (n=16) receiving either synchronous or asynchronous tactile stimulation for 5 min, a previous study revealing the illusion starts in as little as 11 sec (Ehrsson et al, 2004).

Experiment 2: Investigating body ownership using static and animated digital images as visual stimuli.
In this experiment the rubber hand was replaced with a photorealistic image of it captured in its original position and under the lighting conditions used in the previous experiment. A high quality digital image was displayed on a Samsung Galaxy Tab 4 tablet computer with a 10.1 inch TFT display screen and 1280 x 800 resolution enabling the image of the rubber hand to be presented as lifesize in full screen mode. The tablet was laid flat on the table and positioned in the same location as the rubber hand as in Experiment 1 (Figure 2). Synchronous/asynchronous tactile stimulation was as described previously and lasted for 4 minutes.
The experiment involved 38 participants (20 female, age range 18-54) and consisted of three treatments conducted as a repeated measures design compensating for the order of presentation:

Experiment 3: An implementation of the RHI using VR.
The setup for Experiment 3 is shown in Figure 3a and involved 30 participants (7 female, age range 18-51) The participant wearing a HMD was seated at table with their pronated left-hand placed over a marker on a blue card on the table with a webcam mounted above them recording an image of the hand and lower arm as would be seen from their point-of-view. The participant was asked to fixate on that part of the table where the 'virtual rubber hand' was in view through the HMD. The footage from the webcam was processed using computer software (Max/MSP) to manipulate the image of the participant's hand according to parameters set for resolution and time-delay. The 'virtual rubber hand' was located in a fixed position by disabling the head-tracking function in the system enabling it to be calibrated to a real-world location on the table, its location being set for each treatment by placing a marker for the participant's hand and positioning the webcam. In this preliminary experiment the web camera image was degraded to 55% resolution (Figure 3c), a pilot study indicating that at this level of visual quality participants recognised the image as a hand, but not their own hand. At lower resolutions, the image was not recognisable as a hand. The digital manipulation was applied to the hand and paintbrush, with a 3D grid background being chroma-keyed into the stereoscopic image used to create the virtual environment for the Android phone used as the display device in the HMD (Figures 3b and 3c).
As in the original RHI a synchronous and asynchronous condition were used. The synchronous condition involved 5 minutes of tactile stimulation on the back of the participant's left hand with a paintbrush as described previously. The asynchronous condition was the same except for the introduction of a 300 ms delay in the video feed to the HMD. A previous study found temporal delays of less than 300 ms to be critical for multisensory integration to occur and hence for the RHI to operate (Shimada et al, 2009).

Measurement and data collection
In all experiments measurements were performed and data collected in a similar manner to the original RHI. The pointing error, often referred to as proprioceptive drift, is an objective measurement used in the RHI and is obtained by requesting the participant to slide their right index finger along the edge of the table in one smooth movement with their eyes closed until they consider it to be aligned with the index finger of their left hand. This measurement was taken before and after the treatment three times, with a displacement occurring towards the rubber or virtual hand if the illusion was successful (Botvinick and Cohen, 1998). Subjective data were collected after the experiment by means of a semi-structured interview. Participants were encouraged to talk openly about their experience of the experiment and were invited to complete a questionnaire. For Experiments 1 and 3 the questionnaire was the one used in the original RHI experiment (Botvinick and Cohen, 1998). In Experiment 2, one of the treatments included a combination of visuotactile and visuomotor stimuli where participants were asked to replicate the movements of an animated digital image of a rubber hand with their hidden hand. In this case we wanted examine if this treatment was able to induce both a sense of ownership and agency in the participants and chose to use the questionnaire developed by Kalckert and Ehrsson (2012) to measure both ownership (experiencing the rubber or digital rubber hand as their own hand) and agency (experiencing control over the rubber or digital rubber hand). The questionnaire included a set of corresponding control statements to account for task compliance and suggestibility effects.

Preliminary results
The proprioceptive drift measurements from Experiments 1 and 3 were analysed using an independent samples t-test, with the null hypothesis being that there would be no significant difference in proprioceptive drift between the synchronous and asynchronous treatments. The alternate hypothesis being that there would be a significant difference between the two treatments.
The mean drift values for Experiment 1 indicate a 4.1 cm drift towards the rubber hand in the synchronous group, compared to 0.10 cm for the asynchronous group, which when analysed by a two-tailed t-test was found to be statistically significant (t (30) = -2.58, p = 0.015) leading us to reject the null hypothesis.
Experiment 3 was an interpretation of the RHI using a VR hand (Figure 3a). The mean drift results show a 3.0 cm drift towards the VR hand in the synchronous group, compared to a -1.1 cm drift away from the VR hand in the asynchronous group, which was also statistically significant when analysed by a two-tailed t-test (t (28) = -5.62, p < 0.001).
For Experiment 2, a life-size photorealistic image of the rubber hand used in the first experiment was displayed on a tablet computer. This digital rubber hand image was static for the first synchronous and second asynchronous (control) treatments, and then animated for the third treatment which included both synchronous visuotactile and active visuomotor stimulation. The three treatments were applied within subjects and the experiment conducted and analysed as a repeated measures design. Our aim being to examine if synchronous visuotactile stimulation of a 2D digital image of the rubber hand could elicit the RHI, as well as investigating the effect of a combination of synchronous visuotactile stimulation and active visuomotor stimulation with the animated digital hand.
Mean values for proprioceptive drift were 3.2 cm towards the static digital rubber hand image for the synchronous tactile stimulation, and 3.3 cm towards the animated digital rubber hand for the synchronous tactile + visuomotor stimulation treatment. In the asynchronous treatment, drift was 1.1 cm towards the static digital rubber hand. Further statistical analysis revealed the proprioceptive drift data violated the assumption of sphericity (Mauchly's Test, Chi-square (2df) = 10.019, p < 0.05) and therefore a Greenhouse-Geisser correction was used. An analysis of variance (ANOVA) revealed a significant treatment effect (F(1.609, 59.537) = 8.209, p < 0.005). A post-hoc pairwise comparison of means with a Bonferroni correction for multiple comparisons revealed a statistically significant difference at the 5% level (p = 0.05) in proprioceptive drift between the synchronous and asynchronous tactile stimulation of the static digital rubber hand, and also between the asynchronous tactile stimulation of the static digital rubber hand and a combination of synchronous tactile stimulation and active visuomotor stimulation involving an animated digital rubber hand. There was no significant difference in proprioceptive drift between the synchronous static and animated digital rubber hand treatments.
Analysis of the subjective data is currently underway. Examining one of the questionnaire statements: 'I felt as if the rubber hand were my hand' revealed for Experiment 1 (repeat of the original RHI) that 7 of the 13 subjects (54%) from the synchronous group agreed with the above statement, compared to 0 from the asynchronous group. In Experiment 2 which used either a static or animated digital image of the rubber hand as stimuli, 16 out of 38 (42%) for the static treatment and 11 out of 38 subjects (29%) for the animated treatment agreed with the above statement; compared to 3 out of 38 (8%) for the control. However, in for the VR experiment, similar numbers of participants from the synchronous and asynchronous groups felt that the virtual hand belonged to them (8 and 9 out of 15, or 53% and 60% respectively).
Preliminary qualitative analysis of interviews conducted post-experiment suggests participants' experiences during experiments to be more complex than them simply experiencing the rubber hand or the digital rubber hand as their own.

CONCLUSION
In this paper we have described three experiments designed to investigate if the original RHI can be reproduced and extended using digital media. Proprioceptive drift results from Experiment 1 corroborated the findings of the original RHI, demonstrating a successful recreation of the illusion using the same experimental design and support the hypothesis that intermodal matching is sufficient for the RHI to occur (Botvinick and Cohen, 1998).
Proprioceptive drift results from Experiment 2 suggest that our digital interventions were able to produce a similar phenomenon to the original RHI. A treatment involving synchronous tactile stimulation of a static digital image of the rubber hand, and another treatment combining synchronous tactile with active visuomotor stimulation using an animated digital image of the rubber hand produced a statistically significant proprioceptive drift towards the digital rubber hand in comparison to the asynchronous control.
Synchronous visuotactile stimulation of the participant's own hand in a VR experiment produced a statistically significant proprioceptive drift towards the virtual digital hand, and a delay in the video feed of 300 ms to the HMD was sufficient to create asynchrony and break the illusion. Interestingly, in contrast to all the other experiments, the drift moved away from the virtual hand in the asynchronous treatment, which may be worth further investigation.
The RHI has been used as a method to study multisensory integration and the sense of body ownership. The digital implementations of this illusion reported here provide a different approach to exploring body ownership and offer the possibility of better control of some experimental parameters. Moreover, the findings may contribute to the understanding of how we experience digital media and also how the user experience of digital media can be augmented to offer new possibilities when designing mixed reality experiences. For example, using haptics to provide a greater sense of ownership towards digital avatars in order to enhance the degree of immersion experienced in virtual and augmented reality (AR). Given the renewed interest and potential impact for VR and AR applications in training, remote working, and collaboration, these ideas merit further investigation.