Comparing Pupil Dilation, Head Movement, and EEG for Distraction Detection of Drivers

This paper investigates the use of pupil dilation, head movement and EEG for detecting distraction and cognitive load of drivers while performing secondary tasks in an automotive environment. We tracked pupil dilation from Tobii Pro Glasses 2, head movement from Kinect and EEG from Emotive Insight system. We have analyzed data using Fast Fourier Transform, Continuous Wavelet Transform, and Discrete Wavelet Transform for the full-length signal as well as in windows of 1 second for real-time implementation. We investigated detection of distraction and cognitive load from three different conditions - free driving, driving with lane change, driving with lane change and operating secondary task for each participant in a driving simulator. Our results show that the pupil dilation, head yaw, and EEG can detect the increase in cognitive load due to operation of secondary task within a time buffer of 1 second which can be adapted for real-time implementation. We have also found that FFT of Pupil dilation shows significant categorization of normal and distracted states than the categorization by DWT which contrasts with state of the art methods. Finally, we have proposed an expert system to alert drivers utilizing the signal processing analysis.


INTRODUCTION
In recent time, distraction of drivers increases with increase in number of sophisticated interactive systems inside car which may lead to road accidents.NHTSA has reported that operation of any secondary task should not take the participants' eyes-off-road time greater than 2 seconds (Ranney 2013).Automating detection of distraction can be useful for alerting drivers and get them to safe zone.Distraction may happen for events both inside and outside vehicle.However, detecting distraction may not be enough to as the driver can be driving but also thinking about his/her personal stress in life.In such situations, measuring cognitive load or affective state of driver also becomes a necessity.Several research works are going on to detect cognitive load of the driver and trying to categorize between a normal state and a distracted state.Cognitive load is detected by invasive as well as non-invasive methods.Invasive methods include invasive EEG tracker, heart-rate tracker and so on.Non-invasive methods include non-invasive EEG, eye tracker, head movement tracker, face tracker, voice pattern tracker, questionnaire (NASA TLX) and so on.

Researchers
(Afzal 2009; Sezgin 2007) investigated on detecting cognitive states by capturing affective states of drivers through facial expression.In such cases, it becomes challenging to capture and process the video at different conditions of luminance and exposure inside the car due to which the system fails to detect a set of facial feature points.Sometimes the facial expressions of each person fail to correspond to the mapped emotion due to individuality of the person.Despite the problems of occlusion, lighting and pose variation, researchers have results on affective computing (Zeng 2009).Researchers have also explored areas of eye gaze movements (Yoshida 2014;Tokuda 2011), heart rate or skin response (Healey 2011), acoustic features of voice (Boril 2011) for detection of the cognitive state of drivers.The skin response system requires intrusive methods which cause users unnecessary discomfort while driving.Acoustic features can be tracked only when the driver is talking.Researchers (Biswas 2018) designed a study using a driving simulator reported an evidence of detecting the distraction of drivers from the velocity of SI, deviation of yaw from Kinect, deviation of yaw from IMU.
Researchers (Redlich 1908;Westphal 1907) found a relation between physical task demand and pupil dilation.Researchers also found that the change in pupil dilation is related to change in the viewing of angles of the photograph (Hess 1975).Recent researchers (Gavas 2017;Duchowski 2018) have used a metric to detect cognitive load by measuring frequency and power of pupil dilation.Gavas, as well as Duchowski, have used chin rest for the experiment to control head movements which makes the system difficult to realize in real-time situations.Researchers (Marshall 2002;Marshall 2007) found that a sudden hike in pupil dilation corresponds to increase in cognitive load.This sudden hike is found by processing the pupil dilation signal for its coefficients of wavelet transform and calculating a metric called Index of Cognitive Activity (ICA).Marshall has used only mental tasks (questioning the participant to answer vocally) to detect the cognitive activity.Still, there are not many studies on detecting cognitive load from pupil dilation under varying lighting conditions since the pupil dilation is sensitive to variation in surrounding luminance.Researchers have also detected driver's cognitive load by investigating variance in saccadic intrusion, change in fixation duration and blink count (Lee 2007;Liang 2014;Palinko 2010;Yoshida 2014).Toyota (Basir 2004) has a patent for detecting if the driver is looking away from the road by detecting his eyelid movements.Researchers (Prabhakar 2018) worked on using simple commercial off-the-shelf (COTS) sensors like eye gaze tracker, Kinect for operating secondary tasks using multimodal interaction.Usage of such sensors for distraction detection could exploit the sensors' usability both for secondary task interaction and cognitive load detection.We have designed an experiment and evaluated the detection of distraction of drivers related to pupil dilation, head movement (yaw) and EEG in an automotive environment (driving simulator).

RELATED WORK
The human behavior in an environment can be monitored by tracking their hand, head, finger and eye movement.These movements can be tracked using the COTS sensors like Kinect, IMU, eye gaze tracker, LeapMotion trackers, etc.By monitoring this behavior in a car, we can estimate the user's detraction due to eyes off the road or performing any secondary task while driving.But there are situations where drivers do not take their eyes off the road while driving but their thoughts divert them away from the focus on driving.Such distraction makes the driver physically drive the vehicle but mentally unprepared to face risky situations.Such distractions can be detected or estimated by monitoring brain activity.We have used EEG to monitor brain activity for detecting distraction.Researchers (Biswas 2018) have reported that the yaw data from Kinect is statistically significant for normal state and distracted state of the driver.So, we have chosen only the yaw from Kinect for our analysis.Since the temporal lobe of the brain is involved in processing sensory inputs, we have analyzed T7 data of EEG.It also showed better variation than other electrodes.

USER STUDY
We hypothesize that the pupil dilation of drivers can categorize states of their cognition while performing secondary tasks inside a car while driving, into normal state and distracted state.We conducted the following user study to test the hypothesis in the context of a driving simulator.

Participants
A set of 12 participants with an average age of 26 years undertook the study.The female to male ratio was 2:10.All participants were recruited from our university.All students were well versed with driving cars in the driving simulator.We observed the performance of each participant operating the simulator and made sure that the cognitive load due to driving simulator is same for every participant and the difference in cognitive load correspond only to the secondary task.

Material
We used a driving simulator software with ISO 26022 lane changing task and Logitech G29 steering wheel with pedals.We used Tobii Pro Glasses 2 for recording eye gaze and pupil dilation, an Emotive Insight 5 channel wireless EEG tracker for recording EEG and a Microsoft Kinect (Xbox 360) sensor for recording the head movement of participants.The dashboard display is displayed on a Lenovo Yoga 500 laptop.

Design
The study was designed such that each participant had to undergo three trials of driving tasks by wearing Tobii glasses and EEG tracker on their head.

Figure 1: Participant wearing Tobii glasses and EEG tracker and the Kinect placed on the table
There was no traffic on the road in the driving simulator.The assembly of the setup is illustrated in Figure 1.The first trial was to record reference data by letting the participant take a free drive without doing any secondary task.This was taken as reference case (C1).In the second trial, the participant had to drive as well as follow the Lane changing instructions.This trial corresponds to case 2 (C2).In the third trial, the participant had to drive with lane changing instructions as well as perform a secondary task of selecting buttons on the dashboard display in response to an auditory cue.This trial corresponds to case 3 (C3).The dashboard display is mimicked from one of the existing dashboard displays of Jaguar Land Rover.
The dashboard display was displayed to the left of the driving simulator (for right hand driving in India).
To summarise the three conditions: (i) Driving without any secondary tasks (C1) (ii) Driving by following Lane changing instructions (C2) (iii) Driving with Lane change instruction and perform the secondary task of operating a dashboard display (C3)

Fourier Transform
An FFT (Fast Fourier Transform) was performed over the raw data of pupil dilation, head yaw and EEG (T7).The sum of magnitude of single-sided spectrum (SMSS) was calculated for the full-length of the signal.The SMSS of FFT for each participant was compared if the SMSS in C3> SMSS in C2> SMSS in C1.

Wavelet Transform
The raw data of pupil dilation, head yaw and EEG were processed for coefficients of time-frequency components using DWT (Discrete Wavelet Transform) as well as CWT (Continuous Wavelet Transform).If the value was higher than the threshold, it was assigned a binary value 1.If the value was less than the threshold, it was assigned a binary value 0. The total number of such thresholded peaks were counted for each case.We refer this number as MCD (Measure of Cognition due to Distraction).MCD values were compared between cases if MCD of C3> MCD of C2> MCD of C1 for each participant.Each set of data (pupil dilation, head yaw, and EEG) was calculated for MCD values from DWT and CWT for the full-length raw signal as well as a 1-second window (Marshall 2007) of the raw signal for real-time implementation.The zeros in data were removed as it was due to either the inability of the tracker to capture the eyes or the participant blinked his/her eyes.A set of 200 values were trimmed out from the beginning of the data after the calibration of data was done.Amor wavelet function was used for CWT and db8 (Daubechies 8) wavelet function was used for DWT.

Procedure
Participants were instructed to wear the Tobii Pro glasses and EEG tracker.They were instructed to drive smoothly and safely without veering off from the road.In the first trial, the participants were asked to operate the simulator without any other tasks.In the second trial, participants were asked to follow lane changing instructions on the screen and change lanes accordingly.In the third trial, participants were asked to follow lane changing instructions as well as perform a secondary task of selecting buttons by touching on the dashboard display whenever they hear an auditory cue.

Results
Initially, we took the full-length signal of head yaw, pupil dilation and EEG (T7) and performed FFT, DWT and CWT.After we found significant difference between three cases of driving for all sensor data, we performed same analyses for 1second window running over the full-length signal of each sensor data.All the signal processing techniques were carried out using the MATLAB inbuilt functions.The mean SMSS of pupil dilation of left eye data is plotted for each case of driving as shown in Figure 2. A Kruskal-Wallis test found a significant difference (H=19.16,p < 0.01) between mean ranks of at least one pair of groups.Signed rank tests were carried out for three pairs of groups.There was also an evidence (p<0.01) of a difference between pairs C1vsC2 and C1vsC3.There was no significant difference between the pair C2vsC3.A Kruskal-Wallis test did not find any significant difference between the groups for the SMSS in windows of 1 second for pupil dilation of the right eye, head yaw, and EEG.We performed similar analysis for SMSS as well as MCD for pupil dilation of left and right eyes, head yaw and EEG data in 1 second window as well as for full-length signal.Most results using MCD found significant difference between mean ranks of at least one pair of groups.
We have plotted the effect size (η 2 ) of coefficients of head yaw, pupil dilation and EEG corresponding to different tests we analyzed in Figure 3. From the graph, we can see the effect size is higher for different sensors for same test methods.For realtime (1 second window) implementation of the distraction detection system, DWT (MCD) can be chosen for head yaw and EEG whereas FFT (SMSS) can be chosen for pupil dilation.For implementing detection methods in real-time, we thresholded SMSS values per second and MCD values per second.If the value is greater than threshold, it will interpret as a detection and is represented by value 1 and a value 0 for no detection.If the system detects a distraction (value 1), it can alert the driver.Detections happened in a sliding window of 1 second.

Discussion
This study gives a strong evidence that by measuring pupil dilation of drivers, we can categorize the cognitive state of drivers as normal and distracted (operating secondary tasks) within a time buffer of 1 second.Since the wearable glassbased eye tracker helps in providing freedom of head movement, this extends the opportunity for developers to implement the system in real-time.
Though DWT shows higher effect size for head yaw and EEG, we can see that an FFT shows higher effect size than a DWT or CWT for Pupil dilation.It also gives further evidence that the yaw of head movement and the T7 of EEG can also categorize between the two states of cognition of drivers.
The pupil dilation was significant for only the left eye of all participants.This might be because of the design of the secondary task as the dashboard display was placed to the left of the driver (Right Handed Driving).Though the EEG data was significant, the number of positive detections was less compared to that of head yaw and pupil dilation.This might be because of weak contacts made by electrodes on the head of each participant.A better-quality EEG tracker with reliable contacts might give improved results.

REAL-TIME IMPLEMENTATION
After the detection of distraction and cognitive load of the driver, it is another challenging task to alert the driver.The alert can be of auditory, visual or haptic.We have developed an alert system in which the eye gaze tracker detects drowsiness and distraction.We are planning to implement the realtime detection as an expert system.Our proposed system monitors both the system and driver's behaviour and alerts the driver based on the events triggered from driver's behaviour as well as environment.The environment is classified as secondary task operation, talking to passengers/phone, listening to Radio, high speed driving.The driver's behaviour is classified as eye gaze deviation, pupil dilation, head movement and is detected by sensors.The driver's cognitive state is classified as eyes off road, sleeping, stressed (unconscious driving) and excited (rash driving).Based on the triggering of the driver's cognitive states, we categorise the alerts as Beep/Voice Alert when Eyes off Road is True, Sleeping is False, stressed is False, Excited is True, Steering Vibrator/Sound Alert when Eyes off Road is True, Sleeping is True, stressed is False, Excited is False, Voice Alert to take rest when Eyes off Road is False, Sleeping is False, stressed is True, Excited is False.The video in the following URL initially shows the works we did for operating secondary tasks using eye gaze tracking.The latter part of video shows demonstration of the distraction detection and alert system using pupil dilation.URL: https://youtu.be/KYya8--69KY?t=2m25s

CONCLUSION
We have conducted a user study in which participants undertook driving task along with the operation of secondary tasks.Our results show that the pupil dilation, head yaw, and EEG can detect the increase in cognitive load due to operation of secondary task within a time buffer of 1 second which gives the confidence to extend this method to be implemented in real cars.The usage of wearable glass-based eye tracker helps in detection of cognitive activity from pupil dilation without the need of limiting the head movement by a chin-rest.We have also found that an FFT shows better performance in detection than DWT and CWT for pupil dilation.Though the EEG did not give better results, the synergy of head movement (head yaw) and pupil dilation makes the system robust to noise while detecting the increase in the cognitive load of the driver.In the future, we are planning to implement an expert system to detect distraction and integrate with the alert system for real cars as well as aircraft cockpits., S., & Robinson, P. (2009, September)

Figure 2 :
Figure 2: Mean SMSS (in 1 sec windows) of pupil dilation of left eye for three cases