MoCon-VR: Motion Control in Mobile Virtual Realty Applications through Smartwatches

,


INTRODUCTION
Virtual Reality (VR) environments are around from long time; however, recent advancements in mobile and wearable devices make it accessible and intuitive for normal users to use on daily basis (e.g., for watching movies or playing games).The current mobile devices with high processing capabilities make it possible to run the VR technology without any major rendering problem.Therefore, many toolkits (such as at high-end level Oculus VR toolkit and HTC Vive or very low cast Google Cardboard) and 3D VR mobile apps have been launched for the mobile devices.Amongst these, Google Cardboard is an open-source mobile VR platform that is supported by a very simple cardboard-based mobile holder in order to give viewers the desired VR environment in a very cost.Google provides its open-source Cardboard SDK to develop VR apps for three mobile operating systems, i.e., Android, iOS, and Unity.This simple setup encourages the researchers to use it for different purposes, from research to learning, in different application domains.
The mobile VR toolkit is required to be mounted in front of the user's eyes.Mostly users need to move their head or walk in order to interact with the underlying VR environment.For example, for moving left or right users need to move their head in the same direct respectively.However, one of the main drawbacks in such interaction is that users start feeling nauseas, dizzy, or pain in neck if they use it for some longer period of time due to the continuous movement of their head or neck muscles (Holly 2016, Stein 2016, Tussyadiah et al. 2016).In order to tackle this problem, we propose to use smartwatches for interacting and controlling objects inside the 3D mobile VR applications.Most of the current smartwatches already include 3D accelerometer and gyroscope sensors, which make them suitable to use in this context without any additional efforts.
In this work, we utilize the 3D accelerometer sensor in smartwatches as a way to control the movements in a selected open source mobile VR game, called Treasure Hunt, for Cardboard.The main idea is to wear the watch normally and use the wrist movements for controlling the movement of objects inside a mobile VR application.This would enable the users to control the movements inside the mobile VR environments without moving their own heads, which encourage them to use these setups for longer period of time.We also conducted a preliminary study with 10 users to see feedback towards our watch-based control approach in compared to the head-movementbased control approach in VR.Feedback of the 10 participants shows less dizzy and nausea feelings

The Implementation Details
On the smartwatch end, the wrist motion is used as replacement of head motion.MoCon-VR renders eye and camera matrixes at smartwatch end and generates a view matrix that is used in stereo rendering in OpenGL in the underlying mobile.The view is basically a 4*4 matrix obtained by multiplying camera and eye matrix.MoCon-VR framework at smartwatch side sends this matrix to the connected mobile.It uses TCP protocol through WiFi instead of Bluetooth due to the large number of data transfer at high speed rate.On the mobile side, the 3D VR app (e.g., Treasure Hunt) creates the 3D view for the user through OpenGL.In our MoCon-VR framework, the mobile acts as a server for the smartwatch and gets the data from it in order to handle the movements inside the VR application.MoCon-VR uses a queue at mobile side to store the continuous incoming data form the connected smartwatch in order to render the current view without losing any data.

The Motion Controlling
In Google Cardboard, mobile is rotated at an angle of 90 degrees.This puts the Y axis horizontally and Z-axis vertically.In mobile, left/right motion is controlled by the Y-axis and up/down motion is controlled by the Z-axis.Unlike the mobile, the smartwatch is not rotated while wearing by the user.So MoCon-VR smartwatch-side setup compensates the orientation issue by rotating the received X-axis values by the watch's accelerometer and then sends it to the mobile server; however, the axis responsible for left/right and up/down motion (i.e., Y-axis and Z-axis respectively) remains same as on mobile.The rotation is applied on the X-axis by 90 degrees.This gives the same orientation as using mobile for motion.The mapping between the wrist motion wearing the smartwatch and the object motion inside the VR mobile app is controlled by the rotation matrices, as shown in Figure 1.The Ry matrix is used for the left/right motion.The angle range is from 0 to 90 degrees for left or right motion of the controlled object inside the VR.When the user's wrist with smartwatch is in a natural forward form (we call it centre), then the value of Ry is zero degree and the controlled VR object stays at default position (see Fig. 2).Moving arm towards left or right increases the angle from 0 to 90 degrees from the centre point and the controlled VR object moves left or right accordingly.In headmovement based VR, we normally do not move our head more than 90 degrees left or right direction.Therefore, MoCon-VR restricts the further left or right movement of the controlled VR object when the smartwatch's angle reaches to its maximum limit on left or right side (see Fig. 3).

Figure 2:
The object inside VR is at default position when the wrist Ry is 0 and Rx is 50 degrees.

Figure 3:
The object inside VR is restricted to go further left or right when the angle reaches to its maximum limit.
The Rx matrix is used for the up/down motion.However, when the user wrist with smartwatch is in a natural forward form, then the value of Rx is 50 degrees and the controlled VR object stays at default position (i.e., the Fig. 2 position).Here, the up or down motion of the VR object is controlled by moving hand up or down.When the upward motion is triggered then the angle goes up to 90 and produces negative values.In the case for downward motion, the angle drops to zero and then goes negative.So in order to make ensure to remain angle positive, MoCon-VR halts the motion (see Fig. 4) as the value enters towards negative.
After halting the motion, the previous view is kept saved and is displayed until the angels are back in their normal ranges.MoCon-VR manages the diagonal movements using these both matrices; hence, provides the flexibility of moving the VR object diagonally in the VR environment through moving the wrist in the same direction.

Figure 4:
The object inside VR is restricted to go further up or down when the angle reaches to its maximum limit.

THE PRELIMINARY USER STUDY
We conducted a preliminary user study with 10 participants (2 females), aged 23 -27 (M = 25.7), in order to see their reaction and feedback towards our proposed approach of handling movement in VR environments through smartwatch.Six participants had previous VR experience and own some kind of VR gadgets, 8 participants had experience of watching 3D movies in cinema, while 2 participants never had any kind of past VR experience.Two of the participants mentioned that they normally feel motion sickness problem in VR environment.The goal of this preliminary study was to check the reaction of users with our approach in compared to the head-movement-based standard approach.Therefore, we made an exploratory task in which we asked the participants to explore the "Treasure Hunt" game for 2 to 3 minutes.We conducted the experiment using the within-subject strategy, where half of the randomly selected participants first performed the exploratory task using the watch and then using the head, while the other half performed it in the reverse order.Between using the two approaches, we gave 5 minutes break to each user in order to make them relax from the previous usage.
Figure 5 shows the participants' rating for both approaches in closed-ended questionnaire form.Questions 1-4 were related to the easy to control and move objects inside the VR environment through the head/watch.In these cases, the headmovement-based approach received slightly higher score in compared to our watch-based approach.
When we see the participants' feedback in the open-ended questionnaire form, we found out that many participants pointed out about the quick motion of arm.As they were using this interaction mode for the first time and moving arm is quite easy and quick in compared to the head movement.While moving head slowly was easy for them to control the object through the head.
f going from zero to 100, it goes from zero to 90 and then proceeds in negative value.At default osition (figure 5) the value of angle is 50.When upward motion is triggered (figure 8), the angle goes p to 90 and then produces negative value.In the case for downward motion (figure 9), the angle drops zero then goes negative.So here the angle is ensured to remain positive and as soon as the value nters in negative domain, the motion is halted.To halt the motion, the previous view is kept saved and displayed until the angels are back in their normal ranges.f going from zero to 100, it goes from zero to 90 and then proceeds in negative value.At default osition (figure 5) the value of angle is 50.When upward motion is triggered (figure 8), the angle goes p to 90 and then produces negative value.In the case for downward motion (figure 9), the angle drops o zero then goes negative.So here the angle is ensured to remain positive and as soon as the value nters in negative domain, the motion is halted.To halt the motion, the previous view is kept saved and s displayed until the angels are back in their normal ranges.going from zero to 100, it goes from zero to 90 and then proceeds in negative value.At default sition (figure 5) the value of angle is 50.When upward motion is triggered (figure 8), the angle goes to 90 and then produces negative value.In the case for downward motion (figure 9), the angle drops zero then goes negative.So here the angle is ensured to remain positive and as soon as the value ters in negative domain, the motion is halted.To halt the motion, the previous view is kept saved and displayed until the angels are back in their normal ranges. of going from zero to 100, it goes from zero to 90 and then proceeds in negative value.At default position (figure 5) the value of angle is 50.When upward motion is triggered (figure 8), the angle goes up to 90 and then produces negative value.In the case for downward motion (figure 9), the angle drops to zero then goes negative.So here the angle is ensured to remain positive and as soon as the value enters in negative domain, the motion is halted.To halt the motion, the previous view is kept saved and is displayed until the angels are back in their normal ranges.Questions 5-8 were related to their feelings regarding the overall stress, dizzy, nausea, and neck/head muscles tiredness level.In the case of overall stress (i.e., Q. 5), both approaches received nearly the same response.However, participants felt less dizzy and less nausea feelings as well as less neck/head muscles stressing using our watchbased approach.This shows that initially users may need some time to get used to our watch-based approach; however, it enables them to use VR environment for longer time due to less dizzy and nausea feelings.Further, they will not feel much tiring due to less stress on their neck muscles.
In the open-ended questions, participants provided in general favourable feedback towards our watchbased approach.None of the participant rejected the idea of our approach.Three participants preferred the standard head-movement-based approach; while the remaining seven participants preferred our approach mainly due to the flexibility of arm movement and better efficiency than the head-based approach.Nine participants said that they are excited to use our watch-based approach while 8 participants agreed that they would recommend it to others.In the future, we aim to control other kinds of interactions with different objects inside the mobile VR environments through smartwatch.Further, we intend to perform detailed evaluation studies to compare it with the standard head-movement-based approach using complex task scenarios.Also, we plan to see how it can reduce dizzy and nausea feelings amongst children so that it could be used for VR apps to educate children.

Figure 1 :
Figure 1: The rotation matrices for controlling the axes inside the mobile VR environment.

Fig
Fig u re 5 (d e fa u lt p o sitio n )

F
Fig u re 6 (le ft m o tio n )

Fig u re 5 Figure 5 :
Fig u re 5 (d e fa u lt p o sitio n )