Evaluation of the TV Customer Experience Using Eye Tracking Technology

As the TV experience evolves to provide customers with a richer, more interactive experience across multiple devices, it is increasingly important to make the best use of subjective and objective techniques to inform the development of TV user interfaces. This paper describes the design of a new experiment to evaluate the TV customer experience using eye tracking technology, focused on the BT Player, a visually-rich Video-on-Demand application. Eye tracking provides an objective assessment which does not interfere with the natural interaction of the user with the system. The evaluation will capture a unique data set through the observation of test subjects exposed to a prioritised set of test conditions presented within a controlled environment. The paper presents the design of the experiments, including requirements capture, hardware and software setup, experimental protocol, data collection and analysis. The paper also outlines the challenges posed by the dynamic nature of the content and user interaction with the TV interface.


INTRODUCTION
There is growing evidence that the TV experience is evolving from the traditional linear into an ondemand multi-device interactive experience.The emergence of Internet Protocol TV (IPTV) has created a situation where the end devices and platforms used to view video content possess far more processing, data storage and communication capabilities than previously existed.These capabilities allow customers to have more control of their entertainment experience and also provide TV service providers with a much richer source of data on both the preferences of users and the overall operation of their TV services.
However, as the TV customer experience gains more features and inherent complexity, it becomes more important to make the best use of both subjective and objective techniques to evaluate how existing products are used and to enable the creation of new TV experiences which measurably improve customer engagement.Insights gained from such evaluations can also be used to inform the design and analysis of data captured by TV platforms to improve service delivery.This position paper describes the design of a new experiment to evaluate specific aspects of the TV customer experience using eye tracking technology.It is focused on the BT Player, a Videoon-Demand application which is available in the UK to customers of BT"s TV product.The Player can be accessed on the BT TV Set-Top Box, whose primary input device is an infra-red remote control.It enables customers to browse and select from a wide range of entertainment content through a visually-rich interface which is supported by search and recommendation functions.Content may be available as part of a subscription or may be purchased on demand.A range of different user interface metaphors and dynamic elements are employed with the objective of showcasing both content range and application features.
Studies for user evaluation can take many forms from "cognitive walkthroughs", "think-aloud protocols", heuristic evaluation, observational studies, metrics on efficiency and effectiveness, through to questionnaires and system usability scales (SUS) (Brooke, 1996).Eye-tracking has the potential to bring another layer to this evaluation process by providing an objective assessment which does not interfere with the natural interaction of the user with the system.Our evaluation will capture a unique data set through the observation of test subjects exposed to a prioritised set of test conditions presented within a controlled environment.This will be complemented by log data captured by the Set-Top Box during the evaluation and is expected to provide actionable insights on specific aspects of the customer experience.
Other biometrics methods including Electroencephalography (EEG) (Khushaba et al., 2013) and Galvanic Skin Response (GSR) (Bruneau, Sasse, & Mccarthy, 2002) could potentially be combined with eye tracking research to help evaluate participants" engagement with the user interface and content, and may be the subject of future experiments.

EYE TRACKING TECHNOLOGY
Eye tracking offers objective insight into the behaviours and interaction with onscreen material and environmental surroundings.It can give insights into cognitive processes.In particular Just and Carpenter (Just & Carpenter, 1980) introduced the concept of the eye-mind hypothesis.This was in relation to a study on reading comprehension.They worked with assumption that the "eye remains fixated on a word as long as the word is being processed".In general terms, the eye-mind hypothesis makes the assumption that there is a strong correlation between what the user is looking at and cognitive processes.So, by determining where and what a user is looking at it is plausible to speculate what they are interested in.From this hypothesis eye tracking has become a valuable tool in providing objective evaluation of visual user interfaces (Poole & Ball, 2005).Indeed.companies such as Tobii (Tobii, 2015) provide a comprehensive suite of tools to analyse eye gaze metrics for such purposes.Eye tracking is also used for more than evaluation, finding commercial applications on a range of user platforms (Kar & Corcoran, 2017) such as for assistive technology (Tobii Dynavox, 2017) and for virtual reality (Collins, 2016;Feltham, 2017).
A study, reported by Ali-Hasan et al. used eye tracking to assess television and video user experience (Ali-Hasan, Harrington, & Richman, 2008).They found "eye tracking is pivotal in determining complex usability issues that would be difficult to uncover with traditional observational and think-aloud techniques."They also provided best practice guidelines for conducting eye tracking studies.They carried out their evaluation in a lab set up to resemble a living room using the Tobii X120 Eye Tracker, with Tobii Studio software to perform the eye tracking and analysis.They had established a number of research questions and found the eye tracking useful in determining if the users had noticed key onscreen elements, and where their eyes were drawn to most.
Obrist and Bernhaupt (Obrist & Bernhaupt, 2007) used eye tracking to evaluate interactive TV usability for older users.Their study recruited 16 users, 8 aged 50 years and above (also technicallynaïve) and 8 between the ages of 20 and 30 years old.In addition to eye tracking they employed the "thinking aloud" methodology to provide user commentary on any difficulties.At the end of the set tasks they also used a SUS questionnaire and a customised one for user experience to probe further into the user"s interaction.They also gathered metrics on tasks completed successfully and duration of tasks.In terms of eye tracking the study recorded data on the time to first fixation and the rate of fixations.They also used gaze plot images to investigate search activities.This provided a number of relevant insights, for example, that the number of fixations in each Region of Interest (ROI) was "negatively correlated with search efficiency and task efficiency", an observation also reported by (Jacob & Karn, 2003).Furthermore, they concluded that a longer time for the user to settle on the first fixation point indicated difficulty in finding a suitable area to focus on.Their study highlighted that the older and technologynaïve user had on average a longer period to first fixation.
These examples provide some context on how, with a mixture of eye tracking and more standard evaluation approaches, a robust and informative experimental setup and protocol can be developed for the BT Player evaluation.In both of the above studies, the focus was on user interaction and navigation within a player.They were given set tasks such as switching on the player or searching for content.Both examples used eye tracking to determine if key information had been found onscreen.The BT Player evaluation will endeavour to answer some similar design hypotheses, for example, looking at the behaviour in user navigation, uncovering key ROIs, investigating the ability of certain design features to capture the user"s attention, and determining if certain onscreen components are noticed by the user.Unlike the Obrist and Bernhaupt study we will not at this time be reviewing differences in cohort.However, we will be gathering some demographic information and certain user characteristics which may help develop research hypotheses for future studies.

DESIGN
This section describes the design of a first subjective evaluation of the BT Player customer experience using eye tracking technology.

Requirements and Hypotheses
User evaluation is an important part of system development and improvement to ensure that the design is fit for purpose and meets the needs of the user and indeed exceeds their expectations.This due diligence in system design is key to a positive user experience and hence promotes user acceptance and adoption.
Requirements for the evaluation of the BT Player were iteratively developed in discussion with the designers and software developers responsible for the Player and associated products.The requirements were grouped under several categories closely related to key Player functionality and included aspects such as visual elements on the screen, user navigation path and interactive features.
Participant-independent information include gender, age group and previous experience with the BT Player will be collected.Hypotheses include participants usage patterns among various group, as evidenced by the Eye Tracking data, i.e., the median time for participants to find a program which is free to watch among different user groups and whether or not participants would notice a prompting icon when performing the search of a program.Other hypotheses are based on the provided usability requirements on the BT Player.

Hardware and Software
An overview of the experiment setup is presented in Figure 1.The TV is a focal point for the participant with the Eye Tracker placed below it, but not obstructing the view of the TV, more specifically the screen on which the BT Player is presented.The TV and the Eye Tracker are of primary importance providing the stimulus for a participant and collecting participant responses respectively.The rest of the hardware is more specific to this test.The processing and data collection in the test environment is performed on a laptop computer (Dell XPS 13) with connection made to an external video capture device (Magewell USB Capture HDMI Gen 2) to record each session through the combined use of eye tracking software (Tobii Studio) and video recording software (OBS, Open Broadcaster Software).The Eye Tracker itself is connected to the laptop via an Ethernet port.
The Software used in the test environment is comprised of two different programs, Tobii Studio and OBS.Tobii Studio is used with the Eye Tracker for carrying out tests as well as producing some data that will be used for analysis.OBS is used with the Magewell USB Capture HDMI Gen 2 to provide the video stream to the TV through a few steps: the BT TV Set-Top Box provides a video stream via HDMI cable to the Magewell USB Capture HDMI Gen 2, which in turn provides the video stream via USB 3.0 to the Dell XPS 13 laptop through OBS.The video stream is then displayed on the TV, the laptop display extended to the TV for the participant to view.An additional TV can be used to display a Live View of what the participant sees as well as overlaying where users are looking.The additional TV may be of use to the researchers during an experiment.
Figure 1 shows the placement of the participant in seating (at least comfortable if not replicating a home environment) from where they view the TV and interact with the BT Set-top Box via remote control.While the participant views content on the TV the Eye Tracker placed in front of them, not obstructing their view, generates a 3D model of their eyes during a calibration phase to learn and better represent where they are looking on the TV.The adjustable stand allows the Eye Tracker to be placed at the optimal height and position relative to an individual participant to generate the 3D model.

Experimental Protocol
The eye tracking experiments will run in a designated room in BT Riverside Tower building.One session is expected to take about 1 hour.The test procedure will include the short introduction, followed by calibration of eye tracker to the participant, 2-3 simple preliminary tasks for familiarisation with the test environment, the main test consisting of a number of tasks where the participant will be asked to interact with the BT Player and finally the feedback session from the participants on the experience with the evaluation.

Data Collection from the Eye Tracker and Analysis
For the evaluation of visual displays, different eye gaze metrics can be used for different objectives.Ehmke and Wilson (Ehmke, C. & Wilson, 2007) provide a comprehensive review of the different types of eye tracking metrics and what they may indicate during an evaluation.The potential metrics, derived in part from (Ehmke, C. & Wilson, 2007), and highlights which aspects may be relevant to the BT Player evaluation experiments include metrics related to fixation (i.e., time to first fixation, repeat / number of fixations and fixations on ROIs across users ), saccade (i.e., the number of saccades, shifts in direction / backtracks/regressions), scanpath (i.e., long duration / irregularity and transition between ROIs) and gaze (i.e., the number and duration on specific ROI).In addition, analytics techniques will be explored to fuse data captured through eye tracking with data recorded by the BT TV platform itself, including activity logs from the Set Top Box software.This should support the interpretation of the eye tracking data and enable us to determine whether activity logs could potentially provide new insights into the user experience.

CHALLENGES
By the nature of the BT Player, the content and user interaction are dynamic.It is the aim of the project to develop experimental protocols that can maintain user freedom in their interaction with the Player application.Between the changing screens and the different user behaviours, maintaining a set of user evaluation data that can be easily and effectively compared across users is a key goal.Furthermore, this process should be automated as much as possible to enable efficient and replicative analytics of the data.To address these challenges, the project defines scenes of a specific duration that can be compared across users to establish the ROIs.Session data obtained through interaction with the player will be investigated to facilitate automated methods for portioning data into comparable chunks to derive the metrics and statistics on the participant behaviours.
Another challenge is that the experiment will be conducted using the customer-facing BT TV platform, rather than in test environment, so the researchers will have no control over the content options presented within the Player user interface, which could change due to content being regularly added and removed from the library.Therefore two subjects taking part might see different programme or movie selections even if they are given exactly the same task.This could make it difficult to compare two sets of results, but the risk will be mitigated as far as possible by focusing test conditions around content items which are expected to be persistent during the evaluation period.

CONCLUSION
This paper describes the design of a first subjective evaluation of the BT Player customer experience using eye tracking technology.This evaluation will extend best practice by addressing challenges which are unique to a dynamic and visually-rich TV experience.Its results are expected to provide actionable insights on specific aspects of the customer experience, while enabling future experiments to address more complex aspects of the TV customer experience, such as interactions during viewing, and involving multiple devices.

ACKNOWLEDGEMENT
This work is being carried out within the BT Ireland Innovation Centre (BTIIC), a collaborative R&D centre supported by Ulster University and Invest NI.

Figure 1 :
Figure 1: A complete view of the test environment