Virtual Reality Simulation and Eye Tracking to Assess Hazard Perception of Car Drivers

The assessment of a driver’s hazard perception on the road is difficult to measure. Using a Virtual Reality (VR) simulation coupled with eye tracking to objectively detect hazard recognition is a viable technological solution. The objective in this paper is to outline an approach that uses a virtual simulation to assess hazard perception of car drivers. This approach covers the development of a system capable of capturing the data required to interpret a subject’s attention (e.g. head and eye gaze scan path), and a proposed approach to using supervised machine learning algorithms to analyse this dataset and predict driver competency in hazard perception.


INTRODUCTION
Virtual Reality (VR) as a technology has seen a new generation of hardware, software and public attention in recent years.This is due to a rapid decrease in the cost of head-mounted displays (HMDs) and tracking technology, as well as an increase in content availability for the consumer.In enterprise, VR is being used in areas where there is value in using it over traditional methods -for example, simulation-based training.
Simulation-based training using VR has been used in a number of fields including medicine and aerospace.Specifically, for spatial learning in domains that require precise psychomotor skills.Only very recently has VR achieved the capability to reproduce real world scenarios with a high degree of fidelity, in so much that individuals can enhance their motor skills using this technology [1].VR as an approach to motor skills training using simulated environments and digital assets has obvious advantages, due to cost-effectiveness, repeatability of training and scalability [2].Leveraging VR also has advantages when using it in combination with sensors that measure an individual's head, eye, and hand positions.There is an opportunity to use this data with supervised machine learning, to automatically classify activities and perhaps competence.

Learning to drive a car
VR and simulation based training can certainly be used to provide driving lessons and to provide simulation/VR based assessments.The process of learning to drive is a domain that is not particularly scientific.In the UK and elsewhere, driving lessons are conducted on the road where there is a high level of variance based on location, time, weather and also risk levels.This is the same for practical driving tests and it is a challenge to our understanding of the learning process in this domain as well as our understanding of driving competency and what that means.
VR can be used to simulate driving scenarios in hazardous situations.This is a challenge with assessing driving competence in real life scenarios, as there is a safety consideration.In the UK, there was 1,792 reported road deaths, and 24,101 serious injuries from road traffic accidents in 2016.Hence there is always a need to improve how we assess drivers.
In the UK, a hazard perception test is taken as part of the driving theory test.The test consists of 14 videos which are taken from the perspective of a driver in a road environment.Each video contains at least one developing hazard.The subject is required to press a button to signal that they are aware of each hazard.The sooner the button is pressed the better, contributing more to a cumulative points based system.If the subject scores a high enough cumulative score, meaning their average response time to the hazards was sufficiently low, the subject will pass the hazard perception test.
One potential approach to automatically measuring competency in VR simulations is eye tracking.What an individual looks at, is generally related to what they are paying attention to, hence correlating with cognition.There exists a relationship between what an individual is looking at and what that individual is thinking about.This is known as the eye-mind hypothesis [3].This exposes the opportunity to gain an understanding of cognition in systems where contextual eye-tracking data is available.Outside of the domain of driving, there is research that supports this argument and that has went on to analyse eye- tracking data as a means to understand competency [4].This has typically been conducted using supervised machine learning classification algorithms or regression that learns based on training data provided that covers subjects of varied competency levels, e.g.experts and novices.
In this paper we propose a system using VR technology that simulates driving for the purpose of hazard perception testing, which involves collecting eye-tracking data and computing its context, meaning localising a subject's head, eye, and virtual car position to understand what they are looking at and for how long.We propose an experiment using the proposed system to produce a dataset of subjects of varying driving competency that have all engaged in the same driving activities, in the same virtual environment.This is advantageous as it removes the variance of real-world driving and the influence that would have on the data collected.We then propose an experiment using a series of supervised machine learning algorithms to attempt to build a classifier for driving competency using the eye-tracking data. is an opportunity to use this data to build algorithms that help us understand related, but different, aspects of driving attention and the learning process.

System
The VR system was built using the Unity 3D engine.It uses the HTC Vive HMD.The Vive's resolution is 1080 x 1200 pixels per eye, and has a refresh rate of 90Hz.The system utilises VR that has 6 degrees of freedom (6DoF).6DoF is the name given to VR systems where the HMD is capable of rotating in its three axis (pitch, yaw, roll) and moving in its three axis (x,y,z).This is an important factor for spatial learning, and relates directly to fidelity.A previous generation of HMDs are only capable of providing 3DoF (just pitch, yaw, roll), this creates a fundamental difference between the simulation and the real world.This breaks a subject's ability to use their head and its positions to understand tasks spatially, and changes how a subject can observe a task.Figure 3 shows a screen from the system during the "Car not stopping at junction" hazard.
Throughout the simulation, the subject's head position is recorded.This will be recorded using 6 figures; x, y, z, x rotation, y rotation, z rotation.This defines the position of the subject's head, localised to the vehicle, and the direction the subject is facing.The head tracking data, along with data about the virtual environment as well as the position and rotation of the subject's vehicle in that environment allow us to compute the direction of the subject's head along with information about what is in front of them.Figure 4 shows a top-down (above) and pointof-view (below) heat map of the subject's gaze on the left, and shows information about the observed object(s) on the right, including how long the subject has observed each object in each instance.This screen capture is taken from the "Car reversing from a bay" scenario.

Virtual Reality Simulation and Eye
The head tracking data, along with data about the virtual environment as well as the position and rotation of the subject's vehicle in that environment allow us to compute the direction of the subject's head along with information about what is in front of them.Figure 4 shows a top-down (above) and pointof-view (below) heat map of the subject's gaze on the left, and shows information about the observed object(s) on the right, including how long the subject has observed each object in each instance.This screen capture is taken from the "Car reversing from a bay" scenario.
The frequency at which the data will be captured will be driven by the Unity frame rate, which has been observed to be 60hz at a minimum.

System Development
We propose an improvement to the system where the subject's eyes are tracked as well as their head.This will be achieved through integration of the FOVE HMD, which contains built in eye-tracking capability.
The head and eye tracking data, along with data about the virtual environment as well as the position and rotation of the subject's vehicle in that environment allow us to compute what specifically a given subject is looking at, at a given time.The system will capture data and compute information on what is being looked at, on two different scopes.Firstly, the system will report where within the vehicle the subject is looking; e.g.front window, mirrors, dashboard.Secondly, the system will report where outside of the vehicle the subject is looking; e.g.street, traffic light, hazard.The first system will be guaranteed to return a value, whereas for the second system returning null is a valid response -meaning the subject is not currently observing anything outside of the car.

Dataset Creation
We propose a study where subjects are surveyed and tested within the VR system discussed above.The goal of this study is to collect a sufficiently large dataset for analysis and algorithm development.
This study will use subjects of varied driving competency.This is required for future classification work.There is value in collecting other related information from each subject such as age, gender, length of time since passing a driving test if they have.
The system will then report temporal eye-tracking data containing information about what the subject is looking at, and how long it takes for each subject to observe and react to a hazard.This eye-tracking data will then be interpreted to understand fixations, saccades, and smooth pursuits.

Machine Learning Analysis
We propose using a series of supervised machine learning algorithms, with the proposed dataset as training and testing data, to investigate any classifiable characteristics of the subjects based on their eye tracking data -namely competence at hazard perception.
Taraji E. et al. discuss a system of segmenting eye tracking data "between fixation clusters, saccades, and smooth pursuits" [5] specifically in the domain of driving hazard perception.This research is relevant as it can assist in the process of interpreting the proposed dataset.

DISCUSSION
The value of a tool such as this would rely on the validity of driving simulators and their efficacy in (i) preserving a subject's true driving behaviour in the simulation, and (ii) transferring learned behaviour to the real road from the simulation.A validation study [6] has been conducted with results that "provide early support for the relative validity of the driving simulator".

Challenges
A challenge associated with the development of the system, including the integration of eye tracking data capture, involves preserving ground truth temporal data.The requirement for accurate temporal data means the dynamic elements e.g. the subject's vehicle, the hazards, and the other autonomous vehicles should move independently of rate.Additionally, the eye tracking and head tracking data will need to be captured along with temporal data so that a difference in frame rate does not affect the quality of the dataset.To overcome this challenge, it will involve some validation that the adheres to these guidelines, and potentially some development to ensure that temporal data is accurate and independent of the system's frame rate.

Future Work
This driving simulation, or others like it, could be used to explore all types of driving behaviour, outside of hazard perception.Further work could further develop the VR driving simulator to either; conduct some bespoke scenarios for specific studies, or build a system capable of configuration.A configurable system would create a platform on which various studies could be conducted, with the simulation configured to meet their specification.
Further work could explore the efficacy of using a technique that leverages eye-tracking and control of what a subject sees (looking at a screen, wearing a HMD) to involuntarily draw a subject's gaze [7] as a means to change or improve a subject's behaviour.This work would involve (i) integrating the 'subtle gaze direction' technique into the simulator, (ii) designing a method to determine if this technique can improve a subject's hazard perception, and (iii) conduct an experiment to understand if this improved behaviour can be transferred to real world driving.

CONCLUSION
Eye tracking in simulated VR based training has potential to assess proficiency in which eye tracking and or training is difficult in real world scenarios, such as driving.This paper outlines a study to assess the hazard perception of car drivers using a virtual reality simulation and eye tracking.The three main areas of work discussed are: • Combining the current VR simulation system with eye-tracking capable technology to capture eye-tracking data from VR hazard perception.• Building a dataset by conducting a study to capture data from a group of subjects that have varied levels of driving competence.
• Analysis of the dataset for understanding of driving competence.There is significant room or further research using a more sophisticated simulation system, or through the use of new psychological techniques coming from current virtual reality related research.

Figure 1 .
Figure 1.Photo of a user interacting with the VR application.

Figure 2 .Figure 3 .
Figure 2. Diagram describing current systemFigure2shows the main ideas of the system.The simulation places the subject in a virtual environment.The virtual environment localises the subject's position to a vehicle -in this case a car, with the subject in the driver's seat.The vehicle then autonomously drives throughout a road network, consisting of junctions, traffic lights, and other autonomously driven vehicles.Throughout the simulation the vehicle will have hazardous interactions with other vehicles.These consist of three pre-designed situations:• Car not stopping at junction • Car overtaking parked car • Car reversing from bay

3 Figure 4 .
Figure 4. Screenshot of system visualising gaze information