Virtual Reality, a tool for safe testing of user experience in collaborative robotics

Collaborative robots (cobots) could help humans in tasks that are mundane, dangerous or where human contact carries risk, as it has been recently uncovered by the COVID-19 pandemic. Yet, the collaboration between humans and robots is severely limited by the aspects of safety and comfort of human operators. In this paper, we propose the use of virtual reality (VR) as a way to test collaboration with robots in situations that are difficult or even impossible to safely test in real life, such as those where it would be dangerous to perform testing. Using VR as a means to evaluate collaboration with robots would allow collecting human behavioral data, subjective self-reports, and biosignals signifying human comfort, stress and cognitive load during collaboration. The use of VR allows direct porting of results to real robotic control systems. This approach can revolutionize the way we design, train and test cobots, and open up a range of new design applications: from industry, through healthcare to space operations. The naturalistic collaborative and assistive robots will be also useful when human motor control is impaired, whether by disease (like cerebral palsy or paralysis) or damage (like amputation) or for the older adult population.


INTRODUCTION
Motor collaboration between humans is essential for activities ranging from working together at construction sites to performing complex surgeries.This is because the human ability to read the motor intentions of another human is unparalleled: a skilled technician does not need much instruction to hold up an element that the other one is welding; a nurse does not need much guidance when feeding her patient with a spoon.However, situations like the present COVID-19 pandemic reveal threats to this traditional model of collaboration.The contagion risk posed by human contact impacted many economic branches, such as factories or healthcare.While many institutions have rapidly switched to remote work and communication, many others could not do the same, as human contact is required in many industries.News reports quickly became full of examples: mines in Poland, factories in Germany, or nursing homes in Sweden rapidly turned into sites of accelerated viral spread due to unavoidable in-person human interaction.The need for social distancing was not achievable, and supply shortages, discomfort and the incomplete protection offered by safety gear further negatively affected production lines.In situations of severe risk, such as those described above, human activities could (and should) be at least partially replaced by robots.However, even though the use of collaborative robots (cobots) could minimize the risk to humans, the collaboration between humans and robots is still far and away from nearing, let alone matching the collaboration between humans (Towers-Clark, 2019;IFR, 2018).This thus requires further research, especially in the area, where the interaction between humans and robots may represent a risk for the human.This paper proposes that virtual reality (VR) is used as a tool for safe testing of user experience in collaborative robotics.

2
In doing so, we provide a brief overview of the context to outline how we envision user experience testing in VR contributing to the advancement of human-robot collaboration.

HUMAN-ROBOT COLLABORATION, SAFETY AND ACCEPTABILITY
The general idea of human-robot collaboration is not new, and several companies have deployed collaborative robots capable of working at industry lines.Still, any progress in this domain is severely limited by the safety and acceptability of such collaboration (Towers-Clark, 2019).Human safety is a critical factor, however, as industrial robots are often heavy and/or equipped with powerful effectors, they pose physical danger.For this reason, most industrial robots are kept at a distance or inside safety cages (Table 1).This solution is suboptimal for robots that are supposed to help humans perform their tasks since real cooperation assumes that both agents work simultaneously.The basic way of maintaining robot safety is to include a safety button.However, safety buttons are too slow for many situations due to human manual reaction time.Therefore, robots are often additionally fitted with sensors for detecting human presence, contact force, etc (Robots U., 2021).
Still, such sensors are not enough to ensure human safety, as shown by accidents like the one that happened in 2015 at Volkswagen factory, where a worker was crushed by an assembly line robot (Dockrill, 2015) while in the preceding year, over 30 workers were killed by industrial robots in the US alone (Markoff and Miller, 2014).
Table 1 shows the different levels of collaboration with robots at present.Fenced robots are the noncollaborative, most popular ones.Then there are robots that allow for collaboration.Again, their use is usually limited to cases in red bonds, due to safety.Finally, the last two columns denote actual dynamic collaboration.(2016).

Cell
Unlike the presently available robots, the human brain comes equipped with 'computational machinery' specialized in recognizing and predicting actions.The human brain is extremely efficient in recognizing other people's actions, for example their errors or action intentions (Blakemore and Decety, 2001;Eaves et al., 2016;Cruz, Pires and Nunes, 2017).This recognition can build on the human brain's ability to predictively represent actions (Pilacinski, Wallscheid, and Lindner, 2018;Pilacinski and Lindner, 2019), making humans able to rapidly adapt to what the other human does.However, we do not know whether the human brain applies the same predictive processes to nonhuman agents as it does to humans (Martin and Weisberg, 2003;Osiurak, Rossetti and Badets, 2017).This seems to be an important issue to be investigated in human-robot collaboration.
Naturalistic feeling (acceptability) is an important issue in human-machine interaction (HMI) (Moreno-Briseño, Díaz and Campos-Romo, 2010;Gromeier, Koester and Schack, 2017).For example, one could expect that as collaborative robots become more human-like, the quality and efficiency of human interactions with them would steadily increase.However, this is not always trueif robots resemble humans too closely, they are perceived as strange and unpleasant to interact with (MacDorman and Koch, 2009).This effect is called the "uncanny valley" and is not limited to humans: other social primates also show adverse behaviour towards realistic avatars (Steckenfinger and Ghazanfar, 2009).This suggests that the primate brain may have hardwired neural systems allowing for intuitive discriminating of "natural" behaviour.While the "uncanny valley" has been described for social HMI (Kahn et al., 2007), virtually nothing is known about its impact on collaborative motor performance.Likewise, although it was previously reported (Maurice et al., 2017) that humans operating assistive robots perform better if these robots follow human-like movement patterns (e.g. the relationship between curvature and speed), it is not known 3 whether the same applies to scenarios where humans and cobots work autonomously (like while cooperating).
Human actions are predictable in the sense that arm joint configurations define the degrees of freedom of movement, allowing the brain to construct models of the other person's actions based on natural motor repertoire (Spüler and Niethammer, 2015).For observing robot actions, this is less obvious, as robotic arms do not have the default biomechanical design constraints the human arm has and can execute much more complex movements (such as 360-degree rotations).Yet, the correct prediction of other agent's movements is needed for adapting one's own actions and, as such, efficient cooperation.That is why it is important to understand how different robot designs (more or less human-like in terms of appearance and motion) might impact how humans perceive them and how this perception impacts manual collaboration.
The intuitiveness of other agent's actions is of vital importance in situations where human cognitive effort has to be minimal, such as under threat, stress, fatigue or heightened cognitive load.All these scenarios are difficult to test in the natural world, providing severe limitations to user experience testing of cobots.

THE USE OF VIRTUAL REALITY FOR TESTING HUMAN-ROBOT COLLABORATION
A feasible solution would be to test cobots in immersive VR environments, allowing for testing different types of interaction scenarios and virtual cobots, without putting humans at risk.For example, in scenarios where the user is within the reach of a robot arm, VR allows us to collect measures on related skin response or user's movement patterns without the actual risk to humans.
Using VR for testing would allow testing human interactions with different VR models of real cobots, including those popular in industry, such as Baxter or Kinova.Figure 1 shows VR robot models of increased anthropomorphism, from a one arm basic (R0) to an articulated arm (R1), a two arms Baxter (R2), and a humanoid robot (R3).The middle of Figure 1 shows an example VR collaboration scene.

Different measures and testing scenarios
The VR testing tasks proposed require subjects to grab objects handed to them by the robot, or hand an object to the robot, or simultaneously/jointly with cobot reach to a target object (Weistroffer et al., 2013).The VR setting allows recording of skin conductance (through galvanic skin response), heart rate and muscle activity using respective sensors.In addition, subjects' actions can be tracked and users can be wearing a haptic glove providing touch sensation in VR.
While the VR scenarios themselves may portray a range of scenes, from those taking place in a factory to those of an assistive robot in a care facility, in the VR experiments themselves, one can manipulate, for example: • The robot type, ranging from less anthropomorphic (e.g.simple articulated arm) to humanoid, through a range of intermediate options (Figure 1).
• The action type: cobot grabbing object / cobot handing an object / joint reaching to a target object.
• The cobot hand/joint configurations: cobot hand follows a smooth trajectory (human-like) vs. cobot hand follows a complex pattern of rotations and translations not resembling human hand motion.
• The action range: the robot might/might not reach directly to the human user.
• The eye gaze as a cobot feature (such as found on Baxter).
• The subject cognitive load example, more/less noisy scenes, radio messages).
The presence of cobot gaze is especially interesting as it has been included on some cobots, however the informative aspect of gaze on suggesting cobot intentions has not yet been thoroughly explored in research.

Potential of using VR for testing humanrobot collaboration
A VR framework for testing human-robot collaboration would allow an iterative development process, arguably at a reduced cost, since different iterations could be developed and tested before real-world deployment.This way, the study of human comfort with the robot could become a central part of the design.A feedback loop between the development team and users could be created, leading to more agile cycles of design and redesign.This is particularly important as it would allow for the design, implementation and testing of collaboration models at higher levels of abstraction, without requiring to deal with low-level motor control and perceptual issues.Physiological responses recorded online, such as skin conductance level and heart rate variability could be used to further detect stress levels, e.g. the activation of the fight or flight mechanism (Bradley et al., 2001).This, together with participants' self-reports could provide a more in-depth perspective of cobot acceptability than questionnaires alone.For example, a situation where participants' positive self-reports are combined with physiological markers indicating stress would surface a more complex emotional state that could then be further disentangled.Similarly, users' hand movements can indicate the levels of acceptability of motor cooperation with different cobot types.Natural hand velocity profiles for object-oriented movements are single-peak (Morasso, 1981) and the presence of multiple peaks indicates a change of plan, such as that of adapting to cobot movement (e.g.Flash and Henis, 1991).Analysis of velocity profiles is routinely used in motor neuroscience for assessing hand trajectory programming.Hand trajectories -in combination with hand speed, movement duration and precision -can be a good, objective indicator of human motor performance in collaborating with different types of cobots.
Another important cue in human social life is the eye gaze.The company Rethink Robotics included the gaze feature in their Baxter and Sawyer cobots to increase cobot acceptance.As gaze predictively guides own actions (Johansson et al., 2003) and object affordances (Pilacinski et al., 2021), the gaze is also crucial for reading other agents' intentions (Zuberbühler, 2008).However, to our knowledge, the question of how the presence of cobot gaze impacts human movement parameters has not yet been investigated.In this way, whether the human brain relies on gaze in perceiving actions and intentions of non-human agents remains to be determined.
Combining all the different metrics above and using machine learning paradigms to analyze them can help pinpoint subtle effects that UX questionnaires would not be sensitive enough to measure.For example, it is possible to use a questionnaire to ask users about their level of stress/comfort with alternative cobots scenarios, after they have completed a series of tasks.However, the results of those questionnaires would not answer more important and interesting questions, such as: When did stress kick in?When were users most stressed?Were they stressed on the same task for each scenario or did some cause more/less stress?The retrospective nature of questionnaires means that the results that can be collected through them are too coarse-grained to accurately address more precise questions (Lazar et al. 2017).A portfolio of psychophysiological measures affords us the possibility of using more concrete measurements of the state of the human body to accompany postfact questionnaires.Taking this integrative feedback approach and correlating psychophysiological measures with subjective questionnaires would provide us with a much fuller picture of what an ideal cobot scenario would be for a human than that we would get from only the task performance data and subjective responses.This is particularly relevant in situations that could potentially involve risk and safety issues.

Considering other variables (gender and age)
Although evidence for a substantial influence of gender on motor actions and especially collaborative manual behaviour is scarce, men and women differ in their upper arm and hand biomechanics and some visuomotor skills (Moreno-Briseño, Díaz and Campos-Romo, 2010;Gromeier, Koester and Schack, 2017).Previous studies have shown that males were sensitive to the differences between robotic and anthropomorphic movements, while women largely ignored those differences (Abel et al. 2020, Nomura. 2017).For this reason, gender seems to potentially the measured motor efficiency of collaborating with cobots.We expect gender to further impact acceptability, stress and trust in at least some collaboration scenarios.
Similar to gender, age might play an important role in cobot acceptability, due to factors such as experience with technology, visuomotor abilities, etc.The latter seems especially important in the context of assistive robots aimed at the older population, as this group of users seems to value the physical attractiveness and social likeability of robots more than their younger counterparts (Oh et al. 2020).Furthermore, analysis of gaze behavior has shown that while younger people pay attention to several body parts, older adults focus significantly more on the robot face (Oh et al. 2020).In this way, it is possible that the use of eye gaze might increase the cobots' perceived friendliness and likewise the acceptance in a specific age or gender group.

CONCLUSIONS
The use of VR opens up a whole new array of possibilities to safely and quickly test cobot designs and collaboration scenarios without putting humans at the risk of harm.Modern VR technologies allow the integration of a wide variety of sensory modalities to create aware and immersive scenes.This way, testing cobots can be taken beyond the physical constraints of currently available cobot models and real-world settings.Furthermore, the development process can become more efficient by considering human reactions (i.e., psychological and physiological), leading to a more integrative/efficient approach in human robot interaction.

Figure 1 :
Figure 1: VR robot models (top) and VR collaboration scene (middle) developed by the researchers using Unity Game Engine (Unity Technologies, San Francisco, CA, United States).Bottom: hand velocity profiles showing normal vs. perturbed velocities; the latter is an indicator of reach adaptation, change of plan etc.

Table 1 :
Types of collaboration with industrial robots.As the level of collaboration increases (left to right), so does the Requirement for intrinsic safety features vs. external sensors.Source: In IFR Position Paper (2018) adapted from Bauer